Exterior Monitoring for Vehicles

ABSTRACT

Vehicular arrangement for obtaining information about objects exterior to the vehicle includes distance measuring systems fixed to the vehicle along the windshield, an edge or side of the vehicle, each including a camera that obtains images of objects exterior of the vehicle and a processor that determines a distance between the vehicle and objects included in images obtained by the camera and a speed of the objects based on the successive distance determinations. A reactive system is arranged on the vehicle and coupled to the distance measuring systems, and considers distance between the objects and the vehicle and the speed of the objects and reacts accordingly. A position of the distance measuring systems is selected to encompass an area in front of the vehicle, an area behind the vehicle and an area on each side of the vehicle in the fixed fields of view of the distance measuring systems.

This application is a continuation of U.S. patent application Ser. No.13/596,914 filed Aug. 28, 2012, which is a continuation-in-part (CIP) ofU.S. patent application Ser. No. 11/874,749 filed Oct. 18, 2007, nowU.S. Pat. No. 8,255,144, which is:

1. a CIP of U.S. patent application Ser. No. 11/461,619 filed Aug. 1,2006, now U.S. Pat. No. 7,418,346, which claims priority under 35 U.S.C.§119(e) of U.S. provisional patent application Ser. No. 60/711,452 filedAug. 25, 2005, now expired, and is:

-   -   A) a CIP of U.S. patent application Ser. No. 10/822,445 filed        Apr. 12, 2004, now U.S. Pat. No. 7,085,637, which is:        -   1) a CIP of U.S. patent application Ser. No. 10/118,858            filed Apr. 9, 2002, now U.S. Pat. No. 6,720,920, and        -   2) a CIP of U.S. patent application Ser. No. 10/216,633            filed Aug. 9, 2002, now U.S. Pat. No. 6,768,944; and    -   B) a CIP of U.S. patent application Ser. No. 11/028,386 filed        Jan. 3, 2005, now U.S. Pat. No. 7,110,880 which is a CIP of U.S.        patent application Ser. No. 10/822,445 filed Apr. 12, 2004, now        U.S. Pat. No. 7,085,637; and    -   C) a CIP of U.S. patent application Ser. No. 11/034,325 filed        Jan. 12, 2005, now U.S. Pat. No. 7,202,776 which is a CIP of        U.S. patent application Ser. No. 10/822,445 filed Apr. 12, 2004,        now U.S. Pat. No. 7,085,637;

2. a CIP of U.S. patent application Ser. No. 11/464,385 filed Aug. 14,2006, now U.S. Pat. No. 7,629,899, which claims priority under 35 U.S.C.§119(e) of U.S. provisional patent application Ser. No. 60/711,452filedAug. 25, 2005, now expired, and is a CIP of U.S. patent application Ser.No. 11/028,386 filed Jan. 3, 2005, now U.S. Pat. No. 7,110,880, and aCIP of U.S. patent application Ser. No. 11/034,325 filed Jan. 12, 2005,now U.S. Pat. No. 7,202,776;

3. a CIP of U.S. patent application Ser. No. 11/562,730 filed Nov. 22,2006, now U.S. Pat. No. 7,295,925, which is a CIP of U.S. patentapplication Ser. No. 11/034,325 filed Jan. 12, 2005, now U.S. Pat. No.7,202,776;

4. a CIP of U.S. patent application Ser. No. 11/681,817 filed Mar. 5,2007, now U.S. Pat. No. 7,426,437, which is a CIP of U.S. patentapplication Ser. No.11/034,325 filed Jan. 12, 2005, now U.S. Pat. No.7,202,776; and

5. a CIP of U.S. patent application Ser. No. 11/778,127 filed Jul. 16,2007, now U.S. Pat. No. 7,912,645.

All of the above applications are incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates generally to arrangements and methods formonitoring objects around a vehicle.

BACKGROUND OF THE INVENTION

A detailed discussion of background information is set forth in U.S.patent application Ser. Nos. 60/062,729 filed Oct. 22, 1997, nowexpired, Ser. No. 09/177,041 filed Oct. 22, 1998, now U.S. Pat. No.6,370,475, 60/123,882 filed Mar. 11, 1999, now expired, Ser. No.09/523,559 filed Mar. 10, 2000, now abandoned, Ser. No. 09/679,317 filedOct. 4, 2000, now U.S. Pat. No. 6,405,132, Ser. No. 09/909,466 filedJul. 19, 2001, now U.S. Pat. No. 6,526,352, Ser. No. 10/822,445, nowU.S. Pat. No. 7,085,637, Ser. No. 11/028,386, now U.S. Pat. No.7,110,880, and Ser. No. 11/034,325, now U.S. Pat. No. 7,202,776, all ofwhich are incorporated by reference herein.

All of the patents, patent applications, technical papers and otherreferences mentioned below and in the parent applications areincorporated by reference herein in their entirety. No admission is madethat any or all of these references are prior art and indeed, it iscontemplated that they may not be available as prior art wheninterpreting 35 U.S.C. §102 in consideration of the claims of thepresent application.

Definitions of terms used in the specification and claims are also foundin the parent applications.

SUMMARY OF THE INVENTION

A vehicular arrangement for obtaining information about objects exteriorto the vehicle in accordance with the invention includes severaldistance measuring systems fixed to the vehicle along the windshield, anedge or a side of the vehicle, each including a camera configured toobtain images of objects exterior of the vehicle and a processor thatdetermines a distance between the vehicle and objects included in imagesobtained by the camera and a speed of the objects based on thesuccessive distance determinations. A reactive system is arranged on thevehicle and coupled to the distance measuring systems, and considersboth distance between the objects and the vehicle and the speed of theobjects and reacts accordingly. A position of the distance measuringsystems is selected to encompass an area in front of the vehicle, anarea behind the vehicle and an area on each side of the vehicle in thefixed fields of view of the distance measuring systems.

The reactive system may include a system which limits the speed of thevehicle such that the speed of the vehicle can be limited in thepresence of other objects or vehicles within a threshold distance fromthe vehicle, a warning system for providing a warning to a driver of thevehicle about the objects exterior of the vehicle, and/or be configuredto access a map database containing information about vehicular travellanes and consider the information when determining how to react to thepresence of the object. In the latter case, the reactive system may beconfigured to, based on the map database, ignore objects on travel laneswhere there is a physical barrier separating these travel lanes from thetravel lane on which the vehicle is traveling, ignore stationary objectson a side of the travel lane and/or differentiate between stationaryobjects on a side of the travel lane and stationary objects in thetravel lane.

A vehicle includes any of the configurations of the above arrangementwith four distance measuring systems, one at each of the four corners ofthe vehicle, and which may be coupled to one or more common reactivesystems. Two of the distance measuring systems may be arranged aboveheadlights of the vehicle. Another vehicle includes four distancemeasuring systems, one at each of the four sides of the vehicle, andwhich are coupled to one or more common reactive systems. Yet anothervehicle includes four distance measuring systems, one located on or neara roof of the vehicle, and which are coupled to one or more commonreactive systems.

A method for operating a vehicle in consideration of objects exterior tothe vehicle in accordance with the invention includes obtaining imagesof objects exterior of the vehicle by means of a camera of a pluralityof distance measuring systems, each fixed to the vehicle along thewindshield, an edge or a side of the vehicle, a position of the distancemeasuring systems encompassing an area in front of the vehicle, an areabehind the vehicle and an area on each side of the vehicle in the fixedfields of view of the distance measuring systems, and determining usinga processor, a distance between the vehicle and objects included inimages obtained by the camera and a speed of the objects based on thesuccessive distance determinations. The method also includes determiningusing the processor, whether an action must be undertaken to preventcontact between the object and the vehicle based on the objects andtheir distance from the vehicle, and if so undertaking an action toprevent contact between the vehicle and the object.

The step of undertaking an action may entail limiting the speed of thevehicle in the presence of other vehicles determined to be within athreshold distance from the vehicle, and/or providing a warning to adriver of the vehicle about the objects exterior of the vehicle.Determining whether an action must be undertaken to prevent contactbetween the object and the vehicle may entail accessing a map databasecontaining information about vehicular travel lanes and considering theposition of the object relative to a map in the map database,determining, based on the map database, whether there is a physicalbarrier separating travel lanes on which the objects are moving from thetravel lane on which the vehicle is moving, in which case, the methodentails precluding the action from being undertaken when it isdetermined, based on the map database, that there is a physical barrierseparating these travel lanes from the travel lane on which the vehicleis moving, and/or determining, based on the map database, whether theobject is a stationary object on a side of the travel lane, in whichcase, the method entails precluding the action from being undertakenwhen it is determined, based on the map database, that the object is astationary object on a side of the travel lane. Determining whether anaction must be undertaken to prevent contact between the object and thevehicle may additionally or alternatively entail configuring the systemto differentiate, based on the map database, between stationary objectson a side of the travel lane and stationary objects in the travel lane.

BRIEF DESCRIPTION OF THE DRAWINGS

The various hardware and software elements used to carry out theinvention described herein are illustrated in the form of systemdiagrams, block diagrams, flow charts, and depictions of neural networkalgorithms and structures. Preferred embodiments are illustrated in thefollowing figures:

FIG. 1 illustrates the GPS satellite system with the 24 satellitesrevolving around the earth.

FIG. 2 illustrates four GPS satellites transmitting position informationto a vehicle and to a base station which in turn transmits thedifferential correction signal to the vehicle.

FIG. 3 illustrates a Wide Area Differential GPS or WADGPS system withfour GPS satellites transmitting position information to a vehicle andto a base station which in turn transmits the differential correctionsignal to the vehicle.

FIG. 4 is a logic diagram showing the combination of the GPS system andan inertial navigation system.

FIG. 5 is a block diagram of the overall vehicle accident avoidance,warning, and control system and method of the present inventionillustrating system sensors, radio transceivers, computers, displays,input/output devices and other key elements.

FIG. 5A is a block diagram of a representative accident avoidance,warning and control system.

FIG. 6 is a block diagram of an image analysis computer of the type thatcan be used in the accident avoidance or identification system andmethod of this invention.

FIG. 7 illustrates a vehicle traveling on a roadway in a definedcorridor.

FIG. 8 illustrates two adjacent vehicles traveling on a roadway andcommunicating with each other.

FIG. 9 is a schematic diagram illustrating a neural network of the typeuseful in the image analysis computer of FIG. 5.

FIG. 10 is a schematic diagram illustrating the structure of a nodeprocessing element in the neural network of FIG. 9.

FIG. 11 illustrates the use of a Precise Positioning System employingthree micropower impulse radar transmitters, two or three radarreflectors or three RFID tags in a configuration to allow a vehicle toaccurately determine its position.

FIG. 12 a is a flow chart of the method in accordance with the inventionfor preventing run off the road accidents.

FIG. 12 b is a flow chart of the method in accordance with the inventionfor preventing center (yellow) line crossing accidents.

FIG. 12 c is a flow chart of the method in accordance with the inventionfor preventing stoplight running accidents.

FIG. 13 illustrates an intersection with stop signs on the lesser roadwhere there is a potential for a front to side impact and a rear endimpact.

FIG. 14 illustrates a blind intersection with stoplights where there isa potential for a front side to front side impact.

FIG. 15 illustrates an intersection where there is a potential for afront-to-front impact as a vehicle turns into oncoming traffic.

FIG. 16A is a side view of a vehicle equipped with a road-mappingarrangement in accordance with the invention.

FIG. 16B is a front perspective view of a vehicle equipped with theroad-mapping arrangement in accordance with the invention.

FIG. 17 is a schematic perspective view of a data acquisition module inaccordance with the invention.

FIG. 17A is a schematic view of the data acquisition module inaccordance with the invention.

FIG. 18 shows the view of a road from the video cameras in both of thedata acquisition modules.

FIG. 19 shows a variety of roads and vehicles operating on those roadsthat are in communication with a vehicle that is passing through aPrecise Positioning Station.

FIG. 20 is a schematic of the manner in which communications between avehicle and a transmitter are conducted according to some embodiments ofthe invention.

FIGS. 21A and 21B illustrate a preferred embodiment of a laser radar orcamera system mounted at the four corners of a vehicle above theheadlights and tail lights.

FIGS. 22A and 22B illustrate the system of FIGS. 21A and 21B forvehicles on a roadway.

FIGS. 23A and 23B illustrate an alternative mounting location for laserradar or camera units.

FIG. 24 is a schematic illustration of a typical laser radar deviceshowing the scanning or pointing system with simplified optics.

FIG. 25 is a schematic showing a method for avoiding collisions inaccordance with the invention.

FIG. 26 is a schematic of a multi-form communication system inaccordance with the invention.

FIG. 27 is a schematic of a ubiquitous communication system inaccordance with the invention.

DETAILED DISCUSSION OF PREFERRED EMBODIMENTS 1. Vehicle CollisionWarning and Control

According to U.S. Pat. No. 5,506,584, the stated goals of the US DOTIVHS system are:

-   -   improving the safety of surface transportation    -   increasing the capacity and operational efficiency of the        surface transportation system    -   enhancing personal mobility and the convenience and comfort of        the surface transportation system    -   reducing the environmental and energy impacts of the surface        transportation system

The RtZF® (Road to Zero Fatalities®) system in accordance with thepresent invention satisfies all of these goals at a small fraction ofthe cost of prior art systems. The safety benefits have been discussedabove. The capacity increase is achieved by confining vehicles tocorridors where they are then permitted to travel at higher speeds. Thiscan be achieved immediately where carrier phase DGPS is available orwith the implementation of the highway-located precise location systemsas shown in FIG. 11. An improvement is to add the capability for thespeed of the vehicles to be set by the highway or highway controlsystem. This is a simple additional few bytes of information that can betransmitted along with the road edge location map, thus, at very littleinitial cost. To account for the tolerances in vehicle speed controlsystems, the scanning laser radar, or other technology system, whichmonitors for the presence of vehicles without RtZF® is also usable as anadaptive cruise control system. Thus, if a faster moving vehicleapproaches a slower moving vehicle, it will automatically slow down tokeep a safe separation distance from the leading, slower moving vehicle.Although the system is not planned for platooning, that will be theautomatic result in some cases. The maximum packing of vehicles isautomatically obtained and thus the maximum vehicle flow rate is alsoachieved with a very simple system.

For the Intelligent Highway System (ITS) application, some provision isrequired to prevent unequipped vehicles from entering the restrictedlanes. In most cases, a barrier will be required since if an errantvehicle did enter the controlled lane, a serious accident could result.Vehicles would be checked while traveling down the road or at atollbooth, or similar station, that the RtZF® system was in operationwithout faults and with the latest updated map for the region. Onlythose vehicles with the RtZF® system in good working order would bepermitted to enter. The speed on the restricted lanes would be setaccording to the weather conditions and fed to the vehicle informationsystem automatically, as discussed above. Automatic tolling based on thetime of day or percentage of highway lane capacity in use can also beeasily implemented.

For ITS use, there needs to be a provision whereby a driver can signalan emergency, for example, by putting on the hazard lights. This wouldpermit the vehicle to leave the roadway and enter the shoulder when thevehicle speed is below a certain level. Once the driver provides such asignal, the roadway information system, or the network of vehicle-basedcontrol systems, would then reduce the speed of all vehicles in thevicinity until the emergency has passed. This roadway information systemneed not be actually associated with the particular roadway and alsoneed not require any roadway infrastructure. It is a term used here torepresent the collective system as operated by the network of nearbyvehicles and the inter-vehicle communication system. Eventually, theoccurrence of such emergency situations will be eliminated byvehicle-based failure prediction systems such as described in U.S. Pat.No. 5,809,437.

Emergency situations will develop on intelligent highways. It isdifficult to access the frequency or the results of such emergencies.The industry has learned from airbags that if a system is developedwhich saves many lives but causes a few deaths, the deaths will not betolerated. The ITS system, therefore, must operate with a very highreliability, that is approaching “zero fatalities”™. Since the brains ofthe system will reside in each vehicle, which is under the control ofindividual owners, there will be malfunctions and the system must beable to adapt without causing accidents. An alternative is for thebrains to reside on the network providing that the network connection isreliable.

Spacing of the vehicles is the first line of defense. Secondly, eachvehicle with a RtZF® system has the ability to automatically communicateto all adjacent vehicles and thus immediately issue a warning when anemergency event is occurring. Finally, with the addition of a totalvehicle diagnostic system, such as disclosed in U.S. Pat. No. 5,809,437,potential emergencies can be anticipated and thus eliminated with highreliability.

Although the application for ITS envisions a special highway lane andhigh speed travel, the potential exists in the invention to provide alower measure of automatic guidance where the operator can turn controlof the vehicle over to the RtZF® system for as long as theinfrastructure is available. In this case, the vehicle would operate innormal lanes but would retain its position in the lane and avoidcollisions until a decision requiring operator assistance is required.At that time, the operator would be notified and if he or she did notassume control of the vehicle, an orderly stopping of the vehicle, e.g.,on the side of the road, would occur.

For all cases where vehicle steering control is assumed by the RtZF®system, an algorithm for controlling the steering should be developedusing neural networks or neural fuzzy systems. This is especially truefor the emergency cases discussed herein where it is well known thatoperators frequently take the wrong actions and at the least, they areslow to react. Algorithms developed by other non-pattern recognitiontechniques do not, in general, have the requisite generality orcomplexity and are also likely to make the wrong decisions (although theuse of such systems is not precluded in the invention). When thethrottle and breaking functions are also handled by the system, analgorithm based on neural networks or neural fuzzy systems is even moreimportant.

For the ITS, the driver will enter his or her destination so that thevehicle knows ahead of time where to exit. Alternately, if the driverwishes to exit, he merely turns on his turn signal, which tells thesystem and other vehicles that he or she is about to exit the controlledlane.

Neural networks have been mentioned above and since they can play animportant role in various aspects of the invention, a brief discussionis now presented here. FIG. 9 is a schematic diagram illustrating aneural network of the type useful in image analysis. Data representingfeatures from the images from the CMOS cameras 60 are input to theneural network circuit 63, and the neural network circuit 63 is thentrained on this data (see FIG. 6). More specifically, the neural networkcircuit 63 adds up the feature data from the CMOS cameras 60 with eachdata point multiplied by an associated weight according to theconventional neural network process to determine the correlationfunction.

In this embodiment, 141 data points are appropriately interconnected at25 connecting points of layer 1, and each data point is mutuallycorrelated through the neural network training and weight determinationprocess. In some implementations, each of the connecting points of thelayer 1 has an appropriate threshold value, and if the sum of measureddata exceeds the threshold value, each of the connecting points willoutput a signal to the connecting points of layer 2. In other cases, anoutput value or signal will always be outputted to layer 2 withoutthresholding.

The connecting points of the layer 2 comprises 20 points, and the 25connecting points of the layer 1 are appropriately interconnected as theconnecting points of the layer 2. Similarly, each data value is mutuallycorrelated through the training process and weight determination asdescribed above and in neural network texts. Each of the 20 connectingpoints of the layer 2 can also have an appropriate threshold value, ifthresholding is used, and if the sum of measured data exceeds thethreshold value, each of the connecting points will output a signal tothe connecting points of layer 3.

The connecting points of the layer 3 comprises 3 points in this example,and the connecting points of the layer 2 are interconnected at theconnecting points of the layer 3 so that each data is mutuallycorrelated as described above.

The value of each connecting point is determined by multiplying weightcoefficients and summing up the results in sequence, and theaforementioned training process is to determine a weight coefficient Wjso that the value (ai) is a previously determined output.

ai=ΣWj·Xj(j=1to N)+W ₀

wherein Wj is the weight coefficient,

-   -   Xj is the data    -   N is the number of samples and    -   W₀ is bias weight associated with each node.

Based on this result of the training, the neural network circuit 63generates the weights and the bias weights for the coefficients of thecorrelation function or the algorithm.

At the time the neural network circuit 63 has learned from a suitablenumber of patterns of the training data, the result of the training istested by the test data. In the case where the rate of correct answersof the object identification unit based on this test data isunsatisfactory, the neural network circuit 63 is further trained and thetest is repeated. Typically, about 200,000 feature patterns are used totrain the neural network 63 and determine all of the weights. A similarnumber is then used for the validation of the developed network. In thissimple example chosen, only three outputs are illustrated. These canrepresent another vehicle, a truck and a pole or tree. This might besuitable for an early blind spot detector design. The number of outputsdepends on the number of classes of objects that are desired. However,too many outputs can result in an overly complex neural network and thenother techniques such as modular neural networks can be used to simplifythe process. When a human looks at a tree, for example, he or she mightthink “what kind of tree is that?” but not “what kind of tiger is that”.The human mind operates with modular or combination neural networkswhere the object to be identified is first determined to belong to ageneral class and then to a subclass etc. Object recognition neuralnetworks can frequently make use of this principle with a significantsimplification resulting.

In the above example, the image was first subjected to a featureextraction process and the feature data was input to the neural network.In other cases, especially as processing power continues to advance, theentire image is input to the neural network for processing. Thisgenerally requires a larger neural network. Alternate approaches usedata representing the difference between two frames and the input datato the neural network. This is especially useful when a moving object ofinterest is in an image containing stationary scenery that is of nointerest. This technique can be used even when everything is moving byusing the relative speed as a filter to remove unwanted pixel data. Anyvariations are possible and will now be obvious to those skilled in theart. Alternately, this image can be filtered based on range, which willalso significantly reduce the number of pixels to be analyzed.

In another implementation, the scenes are differenced based onillumination. If infrared illumination is used, for example, theillumination can be turned on and off and images taken and thendifferenced. If the illumination is known only to illuminate an objectof interest then such an object can be extracted from the background bythis technique. A particularly useful method is to turn the illuminationon and off for alternate scan lines in the image. Adjacent scan linescan then be differenced and the resulting image sent to the neuralnetwork system for identification.

The neural network can be implemented as an algorithm on ageneral-purpose microprocessor or on a dedicated parallel processingDSP, neural network ASIC or other dedicated parallel or serialprocessor. The processing speed is generally considerably faster whenparallel processors are used and this can also permit the input of theentire image for analysis rather than using feature data. A combinationof feature and pixel data can also be used.

Neural networks have certain known potential problem areas that variousresearchers have attempted to eliminate. For example, if datarepresenting an object that is totally different from those objectspresent in the training data is input to the neural network, anunexpected result can occur which, in some cases, can cause a systemfailure. To solve this and other neural network problems, researchershave resorted to adding in some other computational intelligenceprinciples such as fuzzy logic resulting in a neural-fuzzy system, forexample. As the RtZF® system evolves, such refinements will beimplemented to improve the accuracy of the system. Thus, although pureneural networks are currently being applied to the problem, hybridneural networks such as modular, combination, ensemble and fuzzy neuralnetworks will undoubtedly evolve.

A typical neural network processing element known to those skilled inthe art is shown in FIG. 10 where input vectors, (X1, X2, . . . , Xn)are connected via weighing elements 120 (W1, W2, . . . , Wn) to asumming node 130. The output of node 130 is passed through a nonlinearprocessing element 140, typically a sigmoid function, to produce anoutput signal, Y. Offset or bias inputs 125 can be added to the inputsthrough weighting circuit 128. The output signal from summing node 130is passed through the nonlinear element 140 which has the effect ofcompressing or limiting the magnitude of the output Y.

Neural networks used in the accident avoidance system of this inventionare trained to recognize roadway hazards including automobiles, trucks,animals and pedestrians. Training involves providing known inputs to thenetwork resulting in desired output responses. The weights areautomatically adjusted based on error signal measurements until thedesired outputs are generated. Various learning algorithms may beapplied with the back propagation algorithm with the Delta Bar rule as aparticularly successful method.

2. Accurate Navigation

2.1 GPS

FIG. 1 shows the current GPS satellite system associated with the earthand including 24 satellites 2, each satellite revolving in a specificorbital path 4 around the earth. By means of such a GPS satellitesystem, the position of any object can be determined with varyingdegrees of precision as discussed herein. A similar system will appearwhen the European Galileo system is launched perhaps doubling the numberof satellites.

2.2 DGPS, WAAS, LAAS and Pseudolites

FIG. 2 shows an arrangement of four satellites 2 designated SV₁, SV₂,SV₃ and SV₄ of the GPS satellite system shown in FIG. 1 transmittingposition information to receiver means of a base station 20, such as anantenna 22, which in turn transmits a differential correction signal viatransmitter means associated with that base station, such as a secondantenna 16, to a vehicle 18.

Additional details relating to FIGS. 1 and 2 can be found in U.S. Pat.No. 5,606,506.

FIG. 3 shows an arrangement of four satellites 2 designated SV₁, SV₂,SV₃ and SV₄ of the GPS satellite system as in FIG. 2 transmittingposition information to receivers of base stations 20 and 21, such as anantenna 22, which in turn transmit a differential correction signal viatransmitters associated with that base stations, such as a secondantenna 16, to a geocentric or low earth orbiting (LEO) satellite 30which in turn transmits the differential correction signals to vehicle18. In this case, one or more of the base stations 20,21 receives andperforms a mathematical analysis on all of the signals received from anumber of base stations that cover the area under consideration andforms a mathematical model of the errors in the GPS signals over theentire area. For the continental U.S. or CONUS, for example, a group of13 or more base stations are operated by OMNISTAR® that are distributedaround the country. By considering data from the entire group of suchstations, the errors in the GPS signals for the entire area can beestimated resulting in a position accuracy of about 3 cm over the entirearea. The corrections are then uploaded to the geocentric or low earthorbiting satellite 30, or communicated to the Internet, forretransmission to vehicles on the roadways. In this way, such vehiclesare able to determine their absolute position to within about 3centimeters. This is known as Wide Area Differential GPS or WADGPS. Thewide area corrections can be further corrected when there are additionallocal stations that are not part of the WADGPS system.

It is important to note that future GPS and Galileo satellite systemsplan for the transmission of multiple frequencies for civilian use. Likea lens, the ionosphere diffracts different frequencies by differentamounts and thus the time of arrival of a particular frequency willdepend on the value of that frequency. This fact can be used todetermine the amount that each frequency is diffracted and thus thedelay or error introduced by the ionosphere. Thus with more than onefrequency being emitted by a particular satellite, the equivalent of theDGPS corrections can be determined be each receiver and there is nolonger a need for DGPS, WADGPS, WAAS, LAAS and similar systems.

The WAAS system is another example of WADGPS for use with airplanes. TheU.S. Government estimates that the accuracy of the WAAS system is about1 meter in three dimensions. Since the largest error is in the verticaldirection, the horizontal error is much less.

3. Maps and Mapping

3.1 Maps

All information regarding the road, both temporary and permanent, shouldbe part of the map database, including speed limits, presence of guardrails, width of each lane, width of the highway, width of the shoulder,character of the land beyond the roadway, existence of poles or treesand other roadside objects, exactly where the precise position locationapparatus is located, etc. The speed limit associated with particularlocations on the maps should be coded in such a way that the speed limitcan depend upon the time of day and the weather conditions. In otherwords, the speed limit is a variable that will change from time to timedepending on conditions. It is contemplated that there will be a displayfor various map information which will always be in view for thepassenger and for the driver at least when the vehicle is operatingunder automatic control. Additional user information can thus also bedisplayed such as traffic conditions, weather conditions,advertisements, locations of restaurants and gas stations, etc.

A map showing the location of road and lane boundaries can be easilygenerated using a specially equipped survey vehicle that has the mostaccurate position measurement system available. In some cases, it mightbe necessary to set up one or more temporary local DGPS base stations inorder to permit the survey vehicle to know its position within a fewcentimeters. However, with the improvements in GPS accuracy frommultiple frequencies and from more accurate DGPS systems, accuratemapping can now be done without requiring carrier phase DGPS. Thevehicle would drive down the roadway while operators, using speciallydesigned equipment, sight the road edges and lanes. This would probablybest be done with laser pointers and cameras. Transducers associatedwith the pointing apparatus record the angle of the apparatus and thenby triangulation determine the distance of the road edge or lane markingfrom the survey vehicle. Since the vehicle's position would beaccurately known, the boundaries and lane markings can be accuratelydetermined. It is anticipated that the mapping activity would take placecontinuously such that all roads in a particular state would beperiodically remapped in order to record any changes which were missedby other monitoring systems and to improve the reliability of the mapsby minimizing the chance for human error. Any roadway changes that werediscovered would trigger an investigation as to why they were notrecorded earlier thus adding feedback to the mapping part of theprocess.

The above-described method depends on human skill and attention and thusis likely to result in many errors. A preferred approach is to carefullyphotograph the edge of the road and use the laser pointers to determinethe location of the road lines relative to the pointers and to determinethe slope of the roadway through triangulation. In this case, severallaser pointers would be used emanating from above, below and/or to thesides of the camera. The reduction of the data is then done later usingequipment that can automatically pick out the lane markings and thereflected spots from the laser pointers. One aid to the mapping processis to place chemicals in the line paint that could be identified by thecomputer software when the camera output is digitized. This may requirethe illumination of the area being photographed by an infrared orultraviolet light, for example.

In some cases where the roadway is straight, the survey vehicle couldtravel at moderate speed while obtaining the boundary and lane locationinformation. In other cases, where the road in turning rapidly, morereadings would be required per mile and the survey vehicle would need totravel more slowly. In any case, the required road information can beacquired semi-automatically with the survey vehicle traveling at amoderate speed. Thus, the mapping of a particular road would not requiresignificant time or resources. It is contemplated that a few such surveyvehicles could map all of the interstate highways in the U.S. in lessthan one year. Eventually, it is contemplated that between 50 and 100such vehicles using photogramity techniques would continuously map andremap the Unites States.

3.2 Mapping

An important part of some embodiments of the invention is the digitalmap that contains relevant information relating to the road on which thevehicle is traveling. The digital map usually includes the location ofthe edge of the road, the edge of the shoulder, the elevation andsurface shape of the road, the character of the land beyond the road,trees, poles, guard rails, signs, lane markers, speed limits, etc. asdiscussed elsewhere herein. Additionally, it can contain the signatureas discussed above. This data or information is acquired in a uniquemanner for use in the invention and the method for acquiring theinformation and its conversion to a map database that can be accessed bythe vehicle system is part of this invention. The acquisition of thedata for the maps will now be discussed. It must be appreciated thoughthat the method for acquiring the data and forming the digital map canalso be used in other inventions.

With reference to FIGS. 16A, 16B, 17 and 17A, a mapping vehicle 200 isused and obtains its location from GPS satellites and its correctionsfrom the local differential stations. Such a system is capable ofproviding the 2 cm accuracy desired for the map database. Typically, atleast two GPS receivers 226 are mounted on the mapping vehicle 200. EachGPS receiver 226 is contained within or arranged in connection with arespective data acquisition module 202, which data acquisition modules202 also contain a GPS antenna 204, an accurate inertial measurementunit (IMU) 206, a forward-looking video camera 208, a downward andoutward looking linear array camera 210 and a scanning laser radar 212.The relative position of these components in FIG. 17 is not intended tolimit the invention.

A processor including a printed circuit board 224 is coupled to the GPSreceivers 226, the IMUs 206, the video cameras 208, the linear cameras210 and the scanning laser radars 212(see FIG. 17A). The processor 224receives information regarding the position of the vehicle from the GPSreceivers 226, and optionally the IMUs 206, and the information aboutthe road from both linear cameras 210 or from both laser radars 212, orfrom all of the linear cameras 210 and laser radars 212, and forms theroad map database. Information about the road can also come from one orboth of the video cameras 208 and be incorporated into the map database.

An alternate preferred approach uses a series of 4-6 cameras lookingforward, backward, and one, two or more on each side. In thisconfiguration, the linear cameras and scanning laser radars can beomitted and all relevant information comes from the IMU and GPS withdifferential corrections. The scene may be illuminated with generalillumination which can be in the IR part of the spectrum. In some cases,laser pointers or another form of structured light is also usedprimarily to permit later analysis of various elevation changes,especially at the side of the roadway. The resulting data is analyzedusing photogramity techniques to obtain a fully digital map.

The map database can be of any desired structure or architecture.Preferred examples of the database structure are of the type discussedin U.S. Pat. No. 6,144,338 (Davies) and U.S. Pat. No. 6,247,019(Davies).

The data acquisition modules 202 are essentially identical and each canmount to the vehicle roof on an extension assembly 214 which extendsforward of the front bumper. Extension assembly 214 can include amounting bracket 216 from the roof of the vehicle 200 forward to eachdata acquisition module 210, a mounting bracket 218 extending from thefront bumper upward to each data acquisition module 202 and a crossmounting bracket 220 extending between the data acquisition modules 202for support. Since all of the data acquisition equipment is co-located,its precise location is accurately determined by the IMU, the mountinglocation on the vehicle and the differential GPS system.

The forward-looking video cameras 208 can provide views of the road asshown in FIG. 18. These cameras 208 permit the database team to observethe general environment of the road and to highlight any anomalies. Theyalso permit the reading of traffic signs and other informationaldisplays all of which can be incorporated into the database. The cameras208 can be ordinary color video cameras, high-speed video cameras, wideangle or telescopic cameras, black and white video cameras, infraredcameras, etc. or combinations thereof. In some cases, special filtersare used to accentuate certain features. For example, it has been foundthat lane markers frequently are more readily observable at particularfrequencies, such as infrared. In such cases, filters can be used infront of the camera lens or elsewhere in the optical path to blockunwanted frequencies and pass desirable frequencies. Polarizing lenseshave also been found to be useful in many cases. Natural illuminationcan be used in the mapping process, but for some particular cases,particularly in tunnels, artificial illumination can also be used in theform of a floodlight or spotlight that can be at any appropriatefrequency of the ultraviolet, visual and infrared portions of theelectromagnetic spectrum or across many frequencies. Laser scanners canalso be used for some particular cases when it is desirable toilluminate some part of the scene with a bright spot. In some cases, ascanning laser rangemeter can be used in conjunction with theforward-looking cameras 204 to determine the distance to particularobjects in the camera view. Other geometries of the mapping vehicle arenot excluded from this general description of one simplifiedarrangement.

The video camera system can be used by itself with appropriate softwareas is currently being done by Lamda Tech International Inc. of Waukesha,Wis., to obtain the location of salient features of a road. However,such a method to obtain accurate maps is highly labor intensive andtherefore expensive. The cameras and associated equipment in the presentinvention are therefore primarily used to supplement the linear cameraand laser radar data acquisition systems to be described now. Thishowever is one approach with a preferred alternate approach using four,six or more cameras as described above.

In this approach, the mapping vehicle data acquisition modules willtypically contain both a linear camera and a scanning laser radar,however, for some applications one or the other may be omitted.

The linear camera 210 is a device that typically contains a linear CCD,CMOS or other light sensitive array of, for example, four thousandpixels. An appropriate lens provides a field of view to this camera thattypically extends from approximately the center of the vehicle out tothe horizon. This camera records a one-dimensional picture covering theentire road starting with approximately the center of the lane andextending out to the horizon. This linear array camera 210 thereforecovers slightly more than 90 degrees. Typically, this camera operatesusing natural illumination and produces effectively a continuous pictureof the road since it obtains a linear picture, or column of pixels, fortypically every one-inch of motion of the vehicle. Thus, a completetwo-dimensional panoramic view of the road traveled by the mappingvehicle is obtained. Since there are two such linear camera units, a 180degree view is obtained. This camera will typically record in full colorthus permitting the map database team to have a complete view of theroad looking perpendicular from the vehicle. The view is recorded in asubstantially vertical plane. This camera will not be able to read texton traffic signs, thus the need for the forward-looking cameras 208.Automated software can be used with the images obtained from thesecameras 208, 210 to locate the edge of the road, lane markers, thecharacter of land around and including the road and all areas that anerrant vehicle may encounter. The full color view allows thecharacterization of the land to be accomplished automatically withminimal human involvement.

The scanning laser radar 212 is typically designed to cover a 90 degreeor less scan thus permitting a rotating mirror to acquire at least foursuch scans per revolution. The scanning laser radar 212 can becoordinated or synchronized with the linear camera 210 so that eachcovers the same field of view with the exception that the camera 210typically will cover more than 90 degrees. Scanning laser radar 212 canbe designed to cover more or less than 90 degrees as desired for aparticular installation. The scanning laser radar 212 can operate in anyappropriate frequency from above ultraviolet to the terahertz.Typically, it will operate in the eye-safe portion of the infraredspectrum for safety reasons. The scanning laser radar 212 can operateeither as a pulse-modulated or a tone-modulated laser as is known in theart. If operating in the tone-modulated regime, the laser light will betypically modulated with three or more frequencies in order to eliminatedistance ambiguities. Noise or code modulated radar can also be used.

For each scan, the laser radar 212 provides the distance from thescanner to the ground for up to several thousand points in a verticalplane extending from approximately the center of the lane out to nearthe horizon. This device therefore provides precise distances andelevations to all parts of the road and its environment. The preciselocation of signs that were observed with the forward-looking cameras204, for example, can now be easily and automatically retrieved. Thescanning laser radar therefore provides the highest level of mappingautomation.

Scanning laser radars have been used extensively for mapping purposesfrom airplanes and in particular from helicopters where they have beenused to map portions of railway lines in the US. Use of the scanninglaser radar system for mapping roadways where the radar is mounted ontoa vehicle that is driving the road is believed to be novel to thecurrent assignee.

Ideally, all of the above-described systems are present on the mappingvehicle. Although there is considerable redundancy between the linearcamera and the scanning laser radar, the laser radar operates at oneoptical frequency and therefore does not permit the automaticcharacterization of the roadway and its environment.

As with the forward-looking cameras, it is frequently desirable to usefilters and polarizing lenses for both the scanning laser radar and thelinear camera. In particular, reflections from the sun can degrade thelaser radar system unless appropriate filters are used to block allfrequencies except frequency chosen for the laser radar.

Laser radars are frequently also referred to as ladars and lidars. Allsuch devices that permit ranging to be accomplished from a scanningsystem, including radar, are considered equivalent for the purposes ofthis invention.

4. Precise Positioning

Another important aid as part of some of the inventions disclosed hereinis to provide markers along the side(s) of roadways which can be eithervisual, passive or active transponders, reflectors, or a variety ofother technologies including objects that are indigenous to or near theroadway, which have the property that as a vehicle passes the marker itcan determine the identity of the marker and from a database it candetermine the exact location of the marker. The term “marker” is meantin the most general sense. The signature determined by a continuous scanof the environment, for example, would be a marker if it is relativelyinvariant over time such as, for example, buildings in a city.Basically, there is a lot of invariant information in the environmentsurrounding a vehicle as it travels down a road toward its destination.From time to time, a view of this invariant landscape or information maybe obstructed but it is unlikely that all of it will be during thetravel of a mile, for example. Thus, a vehicle should be able to matchthe signature sensed with the expected one in the map database andthereby obtain a precise location fix. This signature can be obtainedthrough the use of radar or laser radar technologies as reportedelsewhere herein. If laser radar is used, then an IR frequency can bechosen in the eyesafe part of the spectrum. This will permit highertransmitted power to be used which, especially when used with rangegating, will permit the penetration of a substantial distance throughfog, rain or snow. See in particular Section 5 below and for example,Wang Yanli, Chen Zhe, “Scene matching navigation based on multisensorimage fusion” SPIE Vol. 5286 p. 788-793, 2003 and more recently “Backingup GNSS with laser radar & INS, RAIM in the city, antenna phasewind-up”, Inside GNDD July/August 2007.

For the case of specific markers placed on the infrastructure, if threeor more of such markers are placed along a side of the roadway, apassing vehicle can determine its exact location by triangulation. Notethat even with two such markers using radar with distance measuringcapability, the precise position of a vehicle can be determined asdiscussed below in reference to the Precise Positioning System. In fact,if the vehicle is only able to observe a single radar or lidar reflectorand take many readings as the reflector is passed, it can determinequite accurately its position based on the minimum distance reading thatis obtained during the vehicle's motion past the reflector. Although itmay be impractical to initially place such markers along all roadways,it would be reasonable to place them in particularly congested areas orplaces where it is known that a view of one or more of the GPSsatellites is blocked. A variation of this concept will be discussedbelow.

Although initially it is preferred to use the GPS navigationalsatellites as the base technology, the invention is not limited therebyand contemplates using all methods by which the location of the vehiclecan be accurately determined relative to the earth surface. The locationof the roadway boundaries and the location of other vehicles relative tothe earth surface are also to be determined and all relevant informationused in a control system to substantially reduce and eventuallyeliminate vehicle accidents. Only time and continued system developmentwill determine the mix of technologies that provide the most costeffective solution. All forms of information and methods ofcommunication to and between vehicles are contemplated including directcommunication with stationary and moving satellites, communication withfixed earth-based stations using infrared, optical, terahertz, radar,radio and other segments of the electromagnetic spectrum, direct orindirect communication with the internet and inter-vehiclecommunication. Some additional examples follow:

A pseudo-GPS can be delivered from cell phone stations, in place of orin addition to satellites. In fact, the precise location of a cell phonetower need not initially be known. If it monitors the GPS satellitesover a sufficiently long time period, the location can be determined asthe calculated location statistically converges to the exact location.Thus, every cell phone tower could become an accurate DGPS base stationfor very little cost. DGPS corrections can be communicated to a vehiclevia FM radio via a sub-carrier frequency for example. An infrared orradar transmitter along the highway can transmit road boundary locationinformation. A CD-ROM, DVD or other portable mass storage can be used atthe beginning of a controlled highway to provide road boundaryinformation to the vehicle. Finally, it is contemplated that eventuallya satellite will broadcast periodically, perhaps every five minutes, atable of dates covering the entire CONUS that provides the latest updatedate of each map segment. If a particular vehicle does not have thelatest information for a particular region where it is operating, itwill be able to use its cell phone or other communication system toretrieve such road maps perhaps through the Internet or from an adjacentvehicle. Emergency information would also be handled in a similar mannerso that if a tree fell across the highway, for example, all nearbyvehicles would be notified.

To implement map updating, a signal may be directed by the infrared orradar transmitter to the area covered by a segment of the map relatingto the latest update information for that segment in a form receivableby a transmitter on vehicles passing through the area. A processor onthe vehicle receives the signals, analyzes it and determines whether itsmap includes the latest updated map information for the segment in whichthe vehicle is presently located. If not, an update for the vehicle'smap information is downloaded via the transmitter. This embodiment isparticularly advantageous when the transmitter is arranged before asection of road and thus provides vehicles entering the road and inrange of the transmitter with the map data they will subsequently need.

The transmitter which transmits information to the vehicle, whether mapinformation or other information, may be movable and thus would beparticularly useful for roads undergoing construction, subject toclosure or blockage in view of construction or other factors, or forwhich map data is not yet available. In this case, the movable,temporary transmitter would be able to provide map data for the affectedsection of road to vehicles in range of the transmitter. As thetransmitter is moved along the roadway, the information transmitted canbe changed.

One of the possible problems with the RtZF® system described herein isoperation in areas of large cities such as lower Manhattan. In suchlocations, unless there are a plurality of local pseudolites or preciseposition location system installations or the environment signaturesystem is invoked such as with adaptive associative memories asdescribed above, the signals from the GPS satellites can besignificantly blocked. Also, there is frequently a severe multipathproblem in cities. A solution is to use the LORAN system as a backup forsuch locations. The accuracy of LORAN can be comparable to DGPS. Use ofmultiple roadway-located Precise Positioning Systems would be a bettersolution or a complementary solution. Additionally, some locationimprovement can result from application of the SnapTrack system asdescribed in U.S. Pat. No. 5,874,914 and other patents to Krasner ofSnapTrack.

The use of geo-synchronous satellites as a substitute for earth boundbase stations in a DGPS system, with carrier phase enhancements forsub-meter accuracies, is also a likely improvement to the RtZF® systemthat can have a significant effect in urban areas.

It is expected, especially initially, that there will be many holes inthe DGPS or GPS and their various implementations that will leave thevehicle without an accurate means of determining its location. Theinertial navigation system described above will help in filling theseholes but its accuracy is limited to a time period significantly lessthan about an hour and a distance of less than about 50 miles before itneeds correcting. That may not be sufficient to cover the period betweenDGPS availability. It is therefore contemplated that the RtZF® systemwill also make use of low cost systems located along the roadways thatpermit a vehicle to accurately determine its location.

Such a position-determination assistance system would include aplurality of transmitters placed on or alongside a road, with signalsfrom the transmitters being directed to an area in the path of atraveling vehicle to enable the vehicle to determine its position usingthe transmitted signals and information about the position of thetransmitters. Positional information about the transmitters either beingpreviously provided to the vehicle's processor, e.g., from a mapdatabase, or along with the transmission. The transmitters may be agroup of a linked MIR, IR or RF transmitters which direct signals to acommon area through which vehicles pass. Alternatively, the transmittersmay be a group of a plurality of RFID devices, in which case, one ormore interrogators are arranged on the vehicle to cause the RFID devicesto direct signals in response to an interrogation signal from theinterrogator.

One example of such a system would be to use a group of three MicropowerImpulse Radar (MIR) units such as developed by Lawrence LivermoreLaboratory.

A MIR operates on very low power and periodically transmits a very shortspread spectrum radar pulse. The estimated cost of a MIR is less than$10 even in small quantities. If three such MIR transmitters, 151, 152and 153, as shown in FIG. 11, are placed along the highway and triggeredsimultaneously or with a known delay, and if a vehicle has anappropriate receiver system, the time of arrival of the pulses can bedetermined and thus the location of the vehicle relative to thetransmitters determined. The exact location of the point where all threepulses arrive simultaneously would be the point that is equidistant fromthe three transmitters 151, 152, 153 and would be located on the mapinformation. Only three devices are required since only two dimensionsneed to be determined and it is assumed that the vehicle in on the roadand thus the vertical position is known, otherwise four MIRs would berequired. Thus, it would not even be necessary to have the signalscontain identification information since the vehicle would not be so faroff in its position determination system to confuse different locations.By this method, the vehicle would know exactly where it was whenever itapproached and passed such a triple-MIR installation. The MIR triad PPSor equivalent could also have a GPS receiver and thereby determine itsexact location over time as described above for cell phone towers. Afterthe location has been determined, the GPS receiver can be removed. Inthis case, the MIR triad PPS or equivalent could be placed at will andthey could transmit their exact location to the passing vehicles. Analternate method would be to leave the GPS receiver with the PPS time ofarrival of the GPS data from each satellite so that the passing vehiclesthat do not go sufficiently close to the PPS can still get an exactlocation fix. A similar system using RFID tags is discussed below.

Several such readings and position determinations can be made with oneapproach to the MIR installation, the vehicle need not wait until theyall arrive simultaneously. Also the system can be designed so that thesignals never arrive at the same time and still provide the sameaccuracy as long as there is a sufficiently accurate clock on board thevehicle. One way at looking at FIG. 11 is that transmitters 151 and 152fix the lateral position of the vehicle while transmitters 151 and 153fix the location of the vehicle longitudinally. The three transmitters151,152,153 need not be along the edges on one lane but could spanmultiple lanes and they need not be at ground level but could be placedsufficiently in the air so that passing trucks would not block the pathof the radiation from an automobile. Particularly in congested areas, itmight be desirable to code the pulses and to provide more than threetransmitters to further protect against signal blockage or multipath.

The power requirements for the MIR transmitters are sufficiently lowthat a simple photoelectric cell array can provide sufficient power formost if not all CONUS locations. With this exact location information,the vehicle can become its own DGPS station and can determine thecorrections necessary for the GPS. It can also determine the integerambiguity problem and thereby know the exact number of wave lengthsbetween the vehicle and the satellites or between the vehicle and theMIR or similar station. These calculations can be done on vehicle ifthere is a connection to a network, for example. This would beparticularly efficient as the network, once it had made the calculationsfor one vehicle, would have a good idea of the result for another nearbyvehicle and for other vehicles passing the same spot at a differenttime. This network can be an ad-hoc or mesh network or the internetusing WiMAX, for example. Alternately, the information can be broadcastfrom the vehicle.

MIR is one of several technologies that can be used to provide preciselocation determination. Others include the use of an RFID tag that isdesigned in cooperation with its interrogator to provide a distance tothe tag measurement. Such as RFID can be either an active device with aninternal battery or solar charger or a passive device obtaining itspower from an RF interrogation signal to charge a capacitor or aSAW-based tag operating without power. An alternate and preferred systemuses radar or other reflectors where the time of flight can be measured,as disclosed elsewhere herein.

Once a vehicle passes a Precise Positioning Station (PPS) such as theMIR triad described above, the vehicle can communicate this informationto surrounding vehicles. If the separation distance between twocommunicating vehicles can also be determined by the time-of-flight orequivalent method, then the vehicle that has just passed the triad can,in effect, become a satellite equivalent or moving pseudolite. That is,the vehicle sends (such as by reflection so as not to introduce a timedelay) its GPS data from the satellite and the receiving vehicle thengets the same message from two sources and the time difference is thetime of flight. Finally, if many vehicles are communicating theirpositions to many other vehicles along with an accuracy of positionassessment, each vehicle can use this information along with thecalculated separation distances to improve the accuracy of its positiondetermination. In this manner, as the number of such vehicles increases,the accuracy of the entire system increases until an extremely accuratepositioning system for all vehicles results. Such a system, since itcombines many sources of position information, is tolerant of thefailure of any one or even several such sources. Thus, the RtZF® systembecomes analogous to the Internet in that it cannot be shut down and thegoal of perfection is approached. Some of the problems associated withthis concept will be discussed below.

FIG. 11 shows the implementation of the invention using the PrecisePositioning System (PPS) 151, 152, 153, in which a pair of vehicles 18,26 are traveling on a roadway each in a defined corridor delineated bylines 14 and each is equipped with a system in accordance with theinvention and in particular, each is equipped with PPS receivers. Fourversions of the PPS system will now be described. This invention is notlimited to these examples but they will serve to illustrate theprincipals involved.

Vehicle 18 contains two receivers 160,161 for the micropower impulseradar (MIR) implementation of the invention. MIR or ultrawideband (UWB)transmitter devices are placed at locations 151, 152 and 153respectively. They are linked together with a control wire, not shown,or by a wireless connection such that each device transmits a shortradar or RF pulse at a precise timing relative to the others. Thesepulses can be sent simultaneously or at a precise known delay. Vehicle18 knows from its map database the existence and location of the threeMIR transmitters. The transmitters 151,152 and 153 can either transmit acoded pulse or non-coded pulse. In the case of the coded pulse, thevehicle PPS system will be able to verify that the three transmitters151, 152, 153 are in fact the ones that appear on the map database.Since the vehicle will know reasonably accurately its location and it isunlikely that other PPS transmitters will be nearby or within range, thecoded pulse may not be necessary. Two receivers 160 and 161 areillustrated on vehicle 18. For the MIR implementation, only a singlereceiver is necessary since the position of the vehicle will be uniquelydetermined by the time of arrival of the three MIR pulses. A secondreceiver can be used for redundancy and also to permit the vehicle todetermine the angular position of the MIR transmitters as a furthercheck on the system accuracy. This can be done since the relative timeof arrival of a pulse from one of the transmitters 151, 152, 153 can beused to determine the distance to each transmitter and by geometry itsangular position relative to the vehicle 18. If the pulses are coded,then the direction to the MIR transmitters 151, 152, 153 will also bedeterminable.

The micropower impulse radar units require battery power or anotherpower mechanism to operate. Since they may be joined together with awire in order to positively control the timing of the three pulses, asingle battery can be used to power all three units. This battery canalso be coupled with a solar panel to permit maintenance free operationof the system. Since the MIR transmitters use very small amounts ofpower, they can operate for many years on a single battery.

Although the MIR systems are relatively inexpensive, on the order of tendollars each, the installation cost of the system will be significantlyhigher than the RFID and radar reflector solutions discussed next. TheMIR system is also significantly more complex than the RFID system;however, its accuracy can be checked by each vehicle that uses thesystem. Tying the MIR system to a GPS receiver and using the accurateclock on the GPS satellites as the trigger for the sending of the radarpulses can add additional advantages and complexity. This will permitvehicles passing by to additionally accurately set their clocks to be insynchronization with the GPS clocks. Since the MIR system will know itsprecise location, all errors in the GPS signals can be automaticallycorrected and in that case, the MIR system becomes a differential GPSbase station. For most implementations, this added complexity is notnecessary since the vehicle themselves will be receiving GPS signals andthey will also know precisely their location from the triad of MIRtransmitters 151, 152, 153.

A considerably simpler alternate approach to the MIR system describedabove utilizes reflective RFID tags. These tags, when interrogated by aninterrogator type of receiver 160, 161, reflect or retransmit after aknown delay a modified RF signal with the modification being theidentification of the tag. Such tags are described in many patents andbooks on RFID technology and can be produced for substantially less thanone dollar each. The implementation of the RFID system would involve theaccurate placement of these tags on known objects on or in connectionwith infrastructure. These objects could be spots on the highway, posts,signs, sides of buildings, poles, in highway reflectors or structuresthat are dedicated specifically for this purpose. In fact, any structurethat is rigid and unlikely to change position can be used for mountingRFID tags. In downtown Manhattan, building sides, street lights,stoplights, or other existing structures are ideal locations for suchtags. A vehicle 18 approaching a triad of such RFID tags represented by151, 152, 153 would transmit an interrogation pulse from interrogator160 and/or 161. The pulse would reflect off of, or be retransmitted by,each tag within range and the signal would be received by the sameinterrogator(s) 160, 161 or other devices on the vehicle. Once again, asingle interrogator is sufficient. It is important to note that therange to RFID tags is severely limited unless a source of power isprovided. It is very difficult to provide enough power from RF radiationfrom the interrogator for distances much greater than a few feet. Forlonger distances, a power source should be provided which can be abattery, connection to a power line, solar power, energy harvested fromthe environment via vibration, for example, unless the RFID is based onSAW technology. For SAW technology, reading ranges may be somewhatextended. Greater distances can be achieved using reflectors orreflecting antennas.

Electronic circuitry, not shown, associated with the interrogator 160and/or 161 would determine the precise distance from the vehicle to theRFID tag 151, 152, 153 based on the round trip time of flight and anyretransmission delay in the RFID. This will provide the precise distanceto the three RFID tags 151, 152, 153. Once again, a second interrogator161 can also be used, in which case, it could be a receiver only andwould provide redundancy information to the main interrogator 160 andalso provide a second measure of the distance to each RFID tag. Based onthe displacement of the two receivers 160, 161, the angular location ofeach RFID tag relative into the vehicle can be determined providingfurther redundant information as to the position of the vehicle relativeto the tags.

Using the PPS system, a vehicle can precisely determine its locationwithin about two centimeters relative to the MIR, RFID tags or radar andreflectors and since the precise location of these devices haspreviously been recorded on the map database, the vehicle will be ableto determine its precise location on the surface of the earth. With thisinformation, the vehicle will thereafter be able to use the carrier wavephase to maintain its precise knowledge of its location, as discussedabove, until the locks on the satellites are lost. This prediction ofphase relies on the vehicle system being able to predict the phase ofthe signal from a given satellite that is reaching a fixed location suchas the location that the vehicle was in when it was able to determineits position precisely. This requires an accurate knowledge on thesatellite orbits and an accurate clock. Given this information, thevehicle system should be able to determine the phase of a satellitesignal at the fixed location and at its new location and, by comparingthe phase from such a calculation from each satellite, it should be ableto precisely determine its position relative to the fixed location.Errors due to changes in the ionosphere and the vehicle clock accuracywill gradually degrade the accuracy of these calculations. The vehicle18 can broadcast this information to vehicle 26, for example, permittinga vehicle that has not passed through the PPS triad to also greatlyimprove the accuracy with which it knows its position. Each vehicle thathas recently passed through a PPS triad now becomes a differential GPSstation for as long as the satellite locks are maintained assuming aperfect clock on-board the vehicle and a stable ionosphere. Therefore,through inter-vehicle communications, all vehicles in the vicinity canalso significantly improve their knowledge of their position accuracyresulting in a system which is extremely redundant and therefore highlyreliable and consistent with the “Road to Zero Fatalities”™ process.Once this system is operational, it is expected that the U.S. governmentand other governments will launch additional GPS type satellites, eachwith more civilian readable frequencies, or other similar satellitesystems, further strengthening the system and adding further redundancyeventually resulting in a highly interconnected system that approaches100% reliability and, like the Internet, cannot be shut down.

As the system evolves, the problems associated with urban canyons,tunnels, and other obstructions to satellite view will be solved by theplacement of large numbers of PPS stations, or other devices providingsimilar location information.

FIG. 19 shows a variety of roads and vehicles operating on those roadsthat are in communication with a vehicle that is passing through aPrecise Positioning Station (PPS). The communication system used isbased on noise modulated spread spectrum technologies such as describedin papers by Lukin et al. listed in the parent '445 application.Determination of the presence of any of the PPS devices enables thevehicle to know its approximate location which is sufficient fornavigation purposes when the GPS signals are blocked, unreliable orotherwise not useable or the vehicle does not have a GPS receiver.

FIG. 20 shows a schematic of the operation of a communication and/orinformation transmission system and method in accordance with theinvention. Transmitters are provided, for example at fixed locationsand/or in vehicles or other moving objects, and data about eachtransmitter, such as its location and an identification marker, isgenerated at 240. The location of the transmitter is preferably its GPScoordinates as determined, for example, by a GPS-based positiondetermining system (although other position determining systems canalternatively or additionally be used). The data may include, when thetransmitter is a moving vehicle, the velocity, speed, the direction oftravel, the estimated travel path and the destination of the vehicle.The data is encoded at 242 using coding techniques such as thosedescribed above, e.g., phase modulation of distance or time between codetransmissions, phase or amplitude modulation of the code sequencesthemselves, changes of the polarity of the entire code sequence or theindividual code segments, or bandwidth modulation of the code sequence.The coded data is transmitted at 244 using, e.g., noise or pseudo-noiseradar.

Instead of data about each transmitter being generated at 240, generaldata for transmission could also be generated such as road conditioninformation or traffic information.

A vehicle 246 includes an antenna 248 coupled to a control module,control unit, processor or computer 250. The antenna, which can be animager, 248 receives transmissions (waves or wireless signals) includingtransmissions 252 when in range of the transmitters. The processor 250analyzes the transmissions 252. Such analysis may include adetermination as to whether any transmissions are from transmitterswithin a pre-determined area relative to the vehicle, whether anytransmissions are from transmitters situated within a pre-determineddistance from the vehicle, whether any transmissions are fromtransmitters traveling in a direction toward the vehicle's currentposition, whether any transmissions are from transmitters traveling in adirection toward the vehicle's projected position based on its currentposition and velocity, the angle between the transmitter and thevehicle, and any combinations of such determinations. In general, theinitial analysis may be any position-based filtering, location-basedfiltering, and/or motion-based filtering. Other analyses could bewhether any transmissions are from particular transmitters which mightbe dedicated to the transmission of road conditions data, traffic data,map data and the like. Once the processor 250 ascertains a particulartransmission from a transmitter of interest (for operation of thevehicle, or for any other pre-determined purpose), it extracts theinformation coded in the transmission, but preferably does not extractinformation coded in transmission from transmitters which are not ofinterest, e.g., those from transmitters situated at a location outsideof the pre-determined area. It knows the code because the code isprovided by the transmission, i.e., the initial part of the transmission252 a contains data on the location of the transmitter and the code isbased on the location of the transmitter. As such, once the initial partof the transmission 252 a is received and the location of thetransmitter extracted, the code for the remainder of the transmission252 b can be obtained.

In this manner, the extraction of information from radio frequency wavetransmission may be limited based on a threshold determination (a filterof sorts) as to whether the transmission is of potential interest, e.g.,to the operation of the vehicle based on its position, location and/ormotion. To enable this threshold determination from the analysis of thewaves or filtering of information, the initial part of the transmission252 a can be provided with positional or location information about thetransmitter and information necessitated by the information transferringarrangement (communication protocol data) and the remainder of thetransmission 252 b provided with additional information of potentialinterest for operation of the vehicle. The information contained ininitial part of each transmission (or set of waves) is extracted todetermine whether the information in the final part of the transmissionis of interest. If not, the information in the final part of thetransmission is not extracted. This reduces processing time and avoidsthe unnecessary extraction of mostly if not totally irrelevantinformation. An information filter is therefore provided.

Generating the transmission based on a code derived from the position ofthe transmitter, and thus the vehicle or infrastructure in which or towhich it is fixed, provides significant advantages as discussed above.The code required for spread spectrum, UWB or other communicationsystems is thus determined according to the position of the transmitter,and can be accomplished in several different ways, some of which aredisclosed elsewhere herein. However, use of coded transmissions is notrequired in all embodiments of the information transferring method andarrangement.

Further, the antenna 248 serves as a transmitter for transmittingsignals generated by the processor 250. The processor 248 is constructedor programmed to generate transmissions or noise signals based on itslocation, determined by a position determining device 254 in any knownmanner including those disclosed herein, and encode information aboutthe vehicle in the signals. The information may be an identificationmarker, the type of vehicle, its direction, its velocity or speed, itsproposed course, its occupancy, etc. The processor 248 can encode theinformation in the signals in a variety of methods as disclosed above inthe same manner that the data about the transmitter is encoded. Thus,the processor 248 not only interprets the signals and extractsinformation, it also is designed to generate appropriate noise orotherwise coded signals which are then sent from the antenna 248.

5. Radar and Laser Radar Detection and Identification of ObjectsExternal to the Vehicle

5.1 Sensing of non-RtZF® equipped objects

Vehicles with the RtZF® system in accordance with the invention ideallyshould also be able to detect those vehicles that do not have the systemas well as pedestrians, animals, bicyclists, and other hazards that maycross the path of the equipped vehicle.

Systems based on radar have suffered from the problem of being able tosufficiently resolve the images which are returned to be able toidentify the other vehicles, bridges, etc. except when they are close tothe host vehicle. One method used for adaptive cruise control systems isto ignore everything that is not moving. This, of course, leads toaccidents if this were used with the instant invention. The problemstems from the resolution achievable with radar unless the antenna ismade very large or the object is close. Since this is impractical foruse with automobiles, only minimal collision avoidance can be obtainedusing radar.

Optical systems can provide the proper resolution but may requireillumination with a bright light or laser. If the laser is in theoptical range, there is a danger of causing eye damage to pedestrians orvehicle operators. At a minimum, it will be distracting and annoying toother vehicle operators. A laser operating in the infrared part of theelectromagnetic spectrum avoids the eye danger problem, provided thefrequency is sufficiently far from the visible, and, since it will notbe seen, it will not be annoying. If the IR light is sufficientlyintense to provide effective illumination for the host vehicle, it mightbe a source of blinding light for the system of another vehicle.Therefore a method of synchronization may be required. This could takethe form of an Ethernet protocol, for example, where when one vehicledetects a transmission from another then it backs off and transmits at arandom time later. The receiving electronics would then only be activewhen the return signal is expected. Transmission can also besynchronized based on the GPS time and a scheme whereby two nearbyvehicles would transmit at different times. Since the transmissionduration can be very short, since the intensity of the IR can be high ifit is in the eye-safe range, many adjacent vehicles can transmit eachfraction of a second without interfering with each other,

Another problem arises when multiple vehicles are present that transmitinfrared at the same time if there is a desire to obtain distanceinformation from the scene. In this case, each vehicle needs to be ableto recognize its transmission and not be fooled by transmissions fromanother vehicle. This can be accomplished, as discussed above, throughthe modulation scheme. Several such schemes would suffice with apseudo-noise or code modulation as a preferred method for the presentinvention. This can also be accomplished if each vehicle accuratelyknows its position and controls its time of transmission according to analgorithm that time multiplexes transmissions based on the geographicallocation of the vehicle. Thus, if multiple vehicles are sensed in agiven geographical area, they each can control their transmissions basedon a common algorithm that uses the GPS coordinates of the vehicle toset the time slot for transmission so as to minimize interferencebetween transmissions from different vehicles. Other multiplexingmethods can also be used such as FDMA, CDMA or TDMA, any of which can bebased on the geographical location of the vehicles.

Infrared and terahertz also have sufficient resolution so that patternrecognition technologies can be employed to recognize various objects,such as vehicles, in the reflected image as discussed above. infraredhas another advantage from the object recognition perspective. Allobjects radiate and reflect infrared. The hot engine or tires of amoving vehicle in particular are recognizable signals. Thus, if the areaaround a vehicle is observed with both passive and active infrared, moreinformation can be obtained than from radar, for example. Infrared isless attenuated by fog than optical frequencies, although it is not asgood as radar. Infrared is also attenuated by snow but at the properfrequencies it has about five times the range of human sight. Terahertzunder some situations has an effective range of as much as severalhundred times that of human sight. Note, as with radar, Infrared andterahertz can be modulated with noise, pseudonoise, or other distinctivesignal to permit the separation of various reflected signals fromdifferent transmitting vehicles.

5.2 Laser and Terahertz Radar scanning system

Referring to FIG. 25, a digital map 116 can be provided and when thevehicle's position is determined 118, e.g., by a GPS-based system, thedigital map can be used to define the field 122 that the laser orterahertz radar scanner 102 will interrogate.

Note, when the term scanner is used herein, it is not meant to implythat the beam is so narrow as to require a back and forth motion (ascan) in order to completely illuminate an object of interest. To thecontrary, inventions herein are not limited to a particular beamdiameter other than that required for eye safety. Also a scanner may belimited to an angular motion that just covers a vehicle located 100meters, for example, from the transmitting vehicle, which may involve noangular motion of the scanner at all, or to an angular motion thatcovers 90 or more degrees of the space surrounding the transmittingvehicle. Through the use of high-powered lasers and appropriate optics,an eye safe laser beam can be created that is 5 cm in diameter, forexample, with a divergence angle less than one degree. Such an infraredspotlight requires very little angular motion to illuminate a vehicle at100 meters, for example.

Generally herein, when laser radar, or lidar, is used it will also meana system based on terahertz where appropriate. The laser radar or lidarscanner will return information as to distance to an object in thescanned field, e.g., laser beam reflections will be indicative ofpresence of object in path of laser beam 104 and from these reflections,information such as the distance between the vehicle and the object canbe obtained. This will cover all objects that are on or adjacent to thehighway. The laser pulse can be a pixel that is two centimeters or 1meter in diameter at 50 meters, for example and that pixel diameter canbe controlled by the appropriate optical system that can includeadaptive optics and liquid lenses (such as described in “Liquid lenspromises cheap gadget optics”, NewScientist.com news service, Mar. 8,2004).

The scanner should scan the entire road at such a speed that motion ofthe car can be considered insignificant. Alternately, a separate aimingsystem that operates at a much lower speed, but at a speed to permitcompensation for the car angle changes, may be provided. Such an aimingsystem is also necessary due to the fact that the road curves up anddown. Therefore two scanning methods, one a slow, but for large anglemotion and the other fast but for small angles may be required. Thelarge angular system requires a motor drive while the small angularsystem can be accomplished through the use of an acoustic wave system,such as Lithium Niobate (LiNbO₃), which is used to drive a crystal whichhas a large refractive index such as Tellurium dioxide. Other acousticoptical systems can also be used as scanners.

For these systems, frequently some means is needed to stabilize theimage and to isolate it from vehicle vibrations. Several suchstabilization systems have been used in the past and would be applicablehere including a gyroscopic system that basically isolates the imagingsystem from such vibrations and keeps it properly pointed, apiezoelectric system that performs similarly, or the process can beaccomplished in software where the image is collected regardless of thevibration but where the image covers a wider field of view then isnecessary and software is used to select the region of interest.

Alternately, two systems can be used, a radar system for interrogatinglarge areas and a laser radar for imaging smaller areas. Either or bothsystems can be range gated and noise or pseudonoise modulated.

The laser radar scanner can be set up in conjunction with a range gate106 so that once it finds an object, the range can be narrowed so thatonly that object and other objects at the same range, 65 to 75 feet forexample, are allowed to pass to the receiver. In this way, an image of avehicle can be separated from the rest of the scene for identificationby pattern recognition software 108. Once the image of the particularobject has been captured , the range gate is broadened, to about 20 to500 feet for example, and the process repeated for another object. Inthis manner, all objects in the field of interest to the vehicle can beseparated and individually imaged and identified. Alternately, a schemebased on velocity can be used to separate a part of one object from thebackground or from other objects. The field of interest, of course, isthe field where all objects with which the vehicle can potentiallycollide reside. Particular known and mapped features on the highway canbe used as aids to the scanning system so that the pitch and perhapsroll angles of the vehicle can be taken into account.

Once the identity of the object is known, the potential for a collisionbetween the vehicle and that object and/or consequences of a potentialcollision with that object are assessed, e.g., by a control module,control unit or processor 112. If collision is deemed likely,countermeasures are effected 114, e.g., activation of a driver alertsystem and/or activation of a vehicle control system to alter the travelof the vehicle (as discussed elsewhere herein).

Range gates can be achieved as high speed shutters by a number ofdevices such as liquid crystals, garnet films, Kerr and Pockel cells oras preferred herein as described in patents and patent applications of3DV Systems Ltd., Yokneam, Israel including U.S. Pat. No. 6,327,073,U.S. Pat. No. 6,483,094, US2002/0185590, WO98/39790, WO97/01111,WO97/01112 and WO97/01113.

Prior to the time that all vehicles are equipped with the RtZF® systemdescribed above, roadways will consist of a mix of vehicles. In thisperiod, it will not be possible to totally eliminate accidents. It willbe possible to minimize the probability of having an accident however,if a laser radar system similar to that described in U.S. Pat. No.5,529,138 with some significant modifications is used, or thosedescribed more recently in various patents and patent applications ofFord Global Technologies such as U.S. Pat. Nos. 6,690,017 and 6,730,913,and U.S. Pat. Appl. Publ. Nos. 2003/0034462, 2003/0155513 and2003/0036881. It is correctly perceived by Shaw that the dimensions of aradar beam are too large to permit distinguishing various objects whichmay be on the roadway in the path of the instant vehicle. Laser radarprovides the necessary resolution that is not provided by radar. Laserradar as used in the present invention however would acquiresignificantly more data than anticipated by Shaw. Sufficient data infact would be attained to permit the acquisition of a three-dimensionalimage of all objects in the field of view. The X and Y dimensions ofsuch objects would, of course, be determined knowing the angularorientation of the laser radar beam. The longitudinal or Z dimension canbe obtained by such methods as time-of-flight of the laser beam to aparticular point on the object and reflected back to the detector, byphase methods or by range gating. All such methods are describedelsewhere herein and in patents listed above.

At least two methods are available for resolving the longitudinaldimension for each of the pixels in the image. In one method, a laserradar pulse having a pulse width of one to ten nanoseconds, for example,can be transmitted toward the area of interest and as soon as thereflection is received and the time-of-flight determined, a new pulsewould be sent at a slightly different angular orientation. The laser,therefore, would be acting as a scanner covering the field of interest.A single detector could then be used, if the pixel is sufficientlysmall, since it would know which pixel was being illuminated. Thedistance to the reflection point could be determined by time-of-flightthus giving the longitudinal distance to all points in view on theobject.

Alternately, the entire area of interest can be illuminated and an imagefocused on a CCD or CMOS array. By checking the time-of-flight to eachpixel, one at a time, the distance to that point on the vehicle would bedetermined. A variation of this would be to use a garnet crystal as apixel shutter and only a single detector. In this case, the garnetcrystal would permit the illumination to pass through one pixel at atime through to a detector. A preferred method, however, for thisinvention is to use range gating as described elsewhere herein.

Other methods of associating a distance to a particular reflectionpoint, of course, can now be performed by those skilled in the artincluding variations of the above ideas using a pixel mixing device(such as described in Schwarte, R. “A New Powerful Sensory Tool inAutomotive Safety Systems Based on PMD-Technology”, S-TEC GmbHProceedings of the AMAA 2000) or variations in pixel illumination andshutter open time to determine distance through comparison of rangegated received reflected light. In the laser scanning cases, the totalpower required from the laser is significantly less than in the areaillumination design. However, the ability to correctly change thedirection of the laser beam in a sufficiently short period of timecomplicates the scanning design. The system can work approximately asfollows: The entire area in front of the instant vehicle, perhaps asmuch as a full 180 degree arc in the horizontal plane can be scanned forobjects using either radar or laser radar. Once one or more objects hadbeen located, the scanning range can be severely limited to basicallycover that particular object and some surrounding space using laserradar. Based on the range to that object, a range gate can be used toeliminate all background and perhaps interference from other objects. Inthis manner, a very clear picture or image of the object of interest canbe obtained as well as its location and, through the use of a neuralnetwork, combination neural network or optical correlation or otherpattern recognition system, the identity of the object can beascertained as to whether it is a sign, a truck, an animal, a person, anautomobile or other object. The identification of the object will permitan estimate to be made of the object's mass and thus the severity of anypotential collision.

Once a pending collision is identified, this information can be madeavailable to the driver and if the driver ceases to heed the warning,control of the vehicle could be taken from him or her by the system. Theactual usurpation of vehicle control, however, is unlikely initiallysince there are many situations on the highway where the potential for acollision cannot be accurately ascertained. Consequently, this systemcan be thought of as an interim solution until all vehicles have theRtZF® system described above.

To use the laser radar in a scanning mode requires some mechanism forchanging the direction of the emitted pulses of light. Oneacoustic-optic method of using an ultrasonic wave to change thediffraction angle of a Tellurium dioxide crystal is disclosed elsewhereherein. This can also be done in a variety of other ways such as throughthe use of a spinning multifaceted mirror, such as is common with laserscanners and printers. This mirror would control the horizontalscanning, for example, with the vertical scanning controlled though astepping motor or the angles of the different facets of the mirror canbe different to slightly alter the direction of the scan, or by othermethods known in the art. Alternately, one or more piezoelectricmaterials can be used to cause the laser radar transmitter to rotateabout a pivot point. A rotating laser system, such as described in Shawis the least desirable of the available methods due to the difficulty inobtaining a good electrical connection between the laser and the vehiclewhile the laser is spinning at a very high angular velocity. Anotherpromising technology is to use MEMS mirrors to deflect the laser beam inone or two dimensions. A newer product is the Digital Light Processor(DLP) from Texas Instruments which contains up to several million MEMSminors which can be rotated through an angle of up to 12 degrees.Although intended for displays, this device can be used to control thedirection(s) of beams from a laser illuminator. The plus or minus 12degree limitation can be expanded through optics but in itself, it isprobably sufficient. See US published patent application No. 20050278098for more details.

Although the system described above is intended for collision avoidanceor at least the notification of a potential collision, when the roadwayis populated by vehicles having the RtZF® system and vehicles which donot, its use is still desirable after all vehicles are properlyequipped. It can be used to search for animals or other objects whichmay be on or crossing the highway, a box dropping off of a truck forexample, a person crossing the road who is not paying attention totraffic. Motorcycles, bicycles, and other non-RtZF® equipped vehiclescan also be monitored.

One significant problem with some previous collision avoidance systemswhich use radar or laser radar systems to predict impacts with vehicles,is the inability to know whether the vehicle that is being interrogatedis located on the highway or is off the road. In at least one system ofthe present invention, the location of the road at any distance ahead ofthe vehicle would be known precisely from the sub-meter accuracy maps,so that the scanning system can ignore, for example, all vehicles onlanes where there is a physical barrier separating the lanes from thelane on which the subject vehicle is traveling. This, of course, is acommon situation on super highways. Similarly, a parked vehicle on theside of the road would not be confused with a stopped vehicle that is inthe lane of travel of the subject vehicle when the road is curving. Thispermits the subject invention to be used for automatic cruise control.In contrast with radar systems, it does not require that vehicles in thepath of the subject vehicle be moving, so that high speed impacts intostalled traffic can be avoided.

If a system with a broader beam to illuminate a larger area on the roadin front of the subject vehicle is used, with the subsequent focusing ofthis image onto a CCD or CMOS array, this has an advantage of permittinga comparison of the passive infrared signal and the reflection of thelaser radar active infrared. Metal objects, for example appear cold topassive infrared. This permits another parameter to be used todifferentiate metallic objects from non-metallic objects such as foliageor animals such as deer. The breadth of the beam can be controlled andthereby a particular object can be accurately illuminated. With thissystem, the speed with which the beam steering is accomplished can bemuch slower. Both systems can be combined into the maximum amount ofinformation to be available to the system.

Through the use of range gating, objects can be relatively isolated fromthe environment surrounding it other than for the section of highwaywhich is at the same distance. For many cases, a properly trained neuralnetwork or other pattern recognition system can use this data andidentify the objects. An alternate approach is to use the Fouriertransform of the scene as input to the neural network or other patternrecognition system. The advantages of this latter approach are that theparticular location of the vehicle in the image is not critical foridentification. Note that the Fourier transform can be accomplishedoptically and optically compared with stored transforms using a garnetcrystal or garnet films, for example, as disclosed in U.S. Pat. No.5,473,466.

At such time, when the system can take control of the vehicle, it willbe possible to have much higher speed travel. In such cases all vehicleson the controlled roadway will need to have the RtZF® or similar systemas described above. Fourier transforms of the objects of interest can bedone optically though the use of a diffraction system. The Fouriertransform of the scene can then be compared with the library of theFourier transforms of all potential objects and, through a system usedin military target recognition, multiple objects can be recognized andthe system then focused onto one object at a time to determine thedegree of threat that it poses.

Of particular importance is the use of a high powered eye-safe laserradar such as a 30 to 100 watt laser diode in an expanded beam form topenetrate fog, rain and snow through the use of range gating. If aseveral centimeter diameter beam is projected from the vehicle in theform of pulses of from 1 to 10 nanoseconds long, for example, and thereflected radiation is blocked except that from the region of interest,an image can still be captured even though it cannot be seen by thehuman eye. This technique significantly expands the interrogation rangeof the system and, when coupled with the other imaging advantages oflaser radar, offers a competitive system to radar and may in fact renderthe automotive use or radar unnecessary. One method is to use thetechniques described in the patents to 3DV listed above. In one case,for example, if the vehicle wishes to interrogate an area 250 feetahead, a 10 nanosecond square wave signal can be used to control theshutter which is used both for transmission and reception and where theoff period can be 480 nanoseconds. This can be repeated until sufficientenergy has been accumulated to provide for a good image. In thisconnection, a high dynamic range camera may be used such as thatmanufactured by IMS chips of Stuttgart, Germany as mentioned above. Sucha camera is now available with a dynamic range of 160 db. According toIMS, the imager can be doped to significantly increase its sensitivityto IR,

These advantages are also enhanced when the laser radar system describedherein is used along with the other features of the RtZF® system such asaccurate maps and accurate location determination. The forward-lookinglaser radar system can thus concentrate its attention to the knownposition of the roadway ahead rather than on areas where there can be nohazardous obstacles or threatening vehicles.

5.3 Blind Spot Detection

The RtZF® system of this invention also can eliminate the need for blindspot detectors such as discussed in U.S. Pat. No. 5,530,447.Alternately, if a subset of the complete RtZF® system is implemented, asis expected in the initial period, the RtZF® system can be madecompatible with the blind spot detector described in the '447 patent.

One preferred implementation for blind spot monitoring as well as formonitoring other areas near the vehicle is the use of range-gated laserradar using a high power laser diode and appropriate optics to expandthe laser beam to the point where the transmitted infrared energy persquare millimeter is below eye safety limits. Such a system is describedabove

5.4 Anticipatory Sensing-Smart Airbags, Evolution of the System

A key to anticipating accidents is to be able to recognize andcategorize objects that are about to impact a vehicle as well as theirrelative velocity. As set forth herein and in the current assignee'spatents and patent applications , this can best be done using a patternrecognition system such as a neural network, combination neural network,optical correlation system, sensor fusion and related technologies. Thedata for such a neural network can be derived from a camera image butsuch an image can be overwhelmed by reflected light from the sun. Infact, lighting variations in general plague camera-based imagesresulting in false classifications or even no classification.Additionally, camera-based systems are defeated by poor visibilityconditions and, additionally, have interference problems when multiplevehicles have the same system which may require a synchronization takingtime away from the critical anticipatory sensing function.

To solve these problems, imaging systems based on millimeter wave radar,laser radar (lidar) and more recently terahertz radar can be used. Allthree systems generally work for anticipatory sensors since the objectsare near the vehicle where even infrared scanning laser radar in anon-range gated mode has sufficient range in fog. Millimeter wave radaris expensive and to obtain precise images a narrow beam is requiredresulting in large scanning antennas. Laser radar systems are lessexpensive and since the beams are formed using optic technology they aresmaller and easier to manipulate.

When computational power is limited, it is desirable to determine theminimum number of pixels that are required to identify an approachingobject with sufficient accuracy to make the decision to take evasiveaction or to deploy a passive restraint such as an airbag. In onemilitary study for anti-tank missiles, it was found that a total of 25pixels are all that is required to identify a tank on a battlefield. Foroptical occupant detection within a vehicle, thousands of pixels aretypically used. Experiments indicate that by limiting the number ofhorizontal scans to three to five, with on the order of 100 to 300pixels per scan that sufficient information is available to find anobject near to the vehicle and in most cases to identify the object.Once the object has been located then the scan can be confined to theposition of the object and the number of pixels available for analysissubstantially increases. There are obviously many algorithms that can bedeveloped and applied to this problem and it is therefore left to thoseskilled in the art. At least one invention is based on the fact that areasonable number of pixels can be obtained from the reflections ofelectromagnetic energy from an object to render each of the proposedsystems practical for locating, identifying and determining the relativevelocity of an object in the vicinity of a vehicle that poses a threatto impact the vehicle so that evasive action can be taken or a passiverestraint deployed. See the discussion in section 5.5 below for apreferred implementation.

The RtZF® system is also capable of enhancing other vehicle safetysystems. In particular, by knowing the location and velocity of othervehicles, for those cases where an accident cannot be avoided, the RtZF®system will in general be able to anticipate a crash and assessment thecrash severity using, for example, neural network technology. Even witha limited implementation of the RtZF® system, a significant improvementin smart airbag technology results when used in conjunction with acollision avoidance system such as described in Shaw (U.S. Pat. Nos.5,314,037 and 5,529,138) and a neural network anticipatory sensingalgorithm such as disclosed in U.S. Pat. No. 6,343,810 . A furtherenhancement would be to code a vehicle-to-vehicle communication signalfrom RtZF® system-equipped vehicles with information that includes thesize and approximate weight of the vehicle. Then, if an accident isinevitable, the severity can also be accurately anticipated and thesmart airbag tailored to the pending event. Information on the type,size and mass of a vehicle can also be implemented as an RFID tag andmade part of the license plate. The type can indicate a vehicle havingprivileges such as an ambulance, fire truck or police vehicle.

Developments by Mobileye include a method for obtaining the distance toan object and thus the relative velocity. Although this technique hasmany limitations, it may be useful in some implementations of one ormore of the current inventions.

A further recent development is reported in U.S. patent applicationpublication No. 20030154010, as well as other patents and patentpublications assigned to Ford Global Technologies including Ser. Nos.06/452,535, 06/480,144, 06/498,972, 06/650,983, 06/568,754, 06/628,227,06/650,984, 06/728,617, 06/757,611, 06/775,605, 06/801,843, 06/819,991,20030060980, 20030060956, 20030100982, 20030154011, 20040019420,20040093141, 20040107033, 20040111200, and 20040117091. In thedisclosures herein, emphasis has been placed on identifying apotentially threatening object and once identified, the properties ofthe object such as its size and mass can be determined. An inferiorsystem can be developed as described in U.S. patent applicationpublication No. 20030154010 where only the size is determined. Ininventions described herein, the size is inherently determined duringthe process of imaging the object and identifying it. Also, the Fordpatent publications mention the combined use of a radar or a lidar and acamera system. Combined use of radar and a camera is anticipated hereinand disclosed in the current assignee's patents.

Another recent development by the U.S. Air Force uses a high poweredinfrared laser operating at wavelengths greater than 1.5 microns and afocal plane array as is reported in “Three-Dimensional Imaging” in AFRLTechnology Horizons, April 2004. Such a system is probably too expensiveat this time for automotive applications. This development illustratesthe fact that it is not necessary to limit the lidar to the nearinfrared part of the spectrum and in fact, the further that thewavelength is away from the visible spectrum, the higher the powerpermitted to be transmitted. Also, nothing prevents the use of multiplefrequencies as another method of providing isolation from transmissionsfrom vehicles in the vicinity. As mentioned above for timingtransmissions, the GPS system can also be used to control the frequencyof transmission thus using frequency as a method to preventinterference. The use of polarizing filters to transmit polarizedinfrared is another method to provide isolation between differentvehicles with the same or similar systems. The polarization angle can bea function of the GPS location of the vehicle.

It is an express intention of some of the inventions herein to provide asystem that can be used both in daytime and at night. Other systems areintended solely for night vision such as those disclosed in U.S. Pat.No. 6,730,913, U.S. Pat. No. 6,690,017 and U.S. Pat. No. 6,725,139. Useof the direction of travel as a method of determining when to transmitinfrared radiation, as disclosed in these and other Ford Global patentsand patent applications, can be useful but it fails to solve the problemof the transmissions from two vehicles traveling in the same vicinityand direction from receiving reflections from each others'transmissions. If the directional approach is used, then some othermethod is required such as coding the pulses, for example.

U.S. Pat. No. 6,730,913 and U.S. Pat. No. 6,774,367 are representativeof a series of patents awarded to Ford Global Technologies as discussedabove. These patents describe range gating as disclosed in the currentassignee's earlier patents. An intent is to supplement the headlightswith a night vision system for illuminating objects on the roadway inthe path of the vehicle but are not seen by the driver and displayingthese objects in a heads-up display. No attempt is made to locate theeyes of the driver and therefore the display cannot place the objectswhere they would normally be located in the driver's field of view asdisclosed in the current assignee's patents. Experiments have shown thatwithout this feature, a night vision system is of little value and mayeven distract the driver to where his or her ability to operate themotor vehicle is degraded. Other differences in the '913 and '367 systeminclude an attempt to compensate for falloff in illumination due todistance, neglecting a similar and potentially more serious falloff dueto scattering due to fog etc. In at least one of the inventionsdisclosed herein, no attempt is made to achieve this compensation in asystematic manner but rather the exposure is adjusted so that asufficiently bright image is achieved to permit object identificationregardless of the cause of the attenuation. Furthermore, in at least oneembodiment, a high dynamic range camera is used which automaticallycompensates for much of the attenuation and thus permits the minimumexposure requirements for achieving an adequate image. In at least oneof the inventions disclosed herein, the system is used both at night andin the daytime for locating and identifying objects and, in some cases,initiating an alarm or even taking control of the vehicle to avoidaccidents. None of these objects are disclosed in the '913 or '367patents and related patents. Additionally, US20030155513, also part ofthis series of Ford Global patents and applications, describesincreasing the illumination intensity based on distance to the desiredfield of view. In at least one of the inventions disclosed herein, theillumination intensity is limited by eye safety considerations ratherthan distance to the object of interest. If sufficient illumination isnot available on one pulse, additional pulses are provided untilsufficient illumination to achieve an adequate exposure is achieved.

If the laser beam diverges, then the amount of radiation per squarecentimeter illuminating a surface will be a function of the distance ofthat surface from the transmitter. If that distance can be measured,then the transmitted power can be increased while keeping the radiationper square centimeter below the eye safe limits. Using this technique,the amount of radiated power can be greatly increased thus enhancing therange of the system in daylight and in bad weather. A lower power pulsewould precede a high power pulse transmitted in a given direction andthe distance measured to a reflective object would be measured and thetransmitted power adjusted appropriately. If a human begins to intersectthe path of transmission, the distance to the human would be measuredbefore he or she could put his or her eye into the transmission path andthe power can be reduced to remain within the safety standards.

It is also important to point out that the inventions disclosed hereinthat use lidar (laser radar or ladar) can be used in a scanning modewhen the area to be covered is larger that the beam diameter or in apointing mode when the beam diameter is sufficient to illuminate thetarget of interest, or a combination thereof.

It can be seen from the above discussion that the RtZF® system willevolve to solve many safety, vehicle control and ITS problems. Even suchtechnologies as steering and drive by wire will be enhanced by the RtZF®system in accordance with invention since it will automatically adjustfor failures in these systems and prevent accidents.

5.5 A Preferred Implementation

FIGS. 21A and 21B illustrate a preferred embodiment of a laser radar orcamera based system having components mounted at the four corners of avehicle above the headlights and tail lights. Although hereinafter laserradar is used, there is also an equivalent optical imager based systemthat can also be used. Laser radar units or assemblies 260 and 261 havea scan angle of approximately 150 degrees; however, for someapplications a larger or smaller scanning angle can of course be used.The divergence angle for the beam for one application can be one degreeor less when it is desired to illuminate an object at a considerabledistance from the vehicle such as from less than fifty meters to 200meters or more. In other cases, where objects are to be illuminated thatare closer to the vehicle, a larger divergence angle can be used.Generally, it is desirable to have a field of illumination (FOI)approximately equal to the field of view (FOV) of the camera or otheroptical receiver. FIGS. 22A and 22B illustrate the system of FIGS. 21Aand 21B for vehicles on a roadway. Note that the divergence angle in thehorizontal plane and vertical plane are not necessarily equal.

FIGS. 23A and 23B illustrate an alternative mounting location for laserradar units on or near the roof of a vehicle. They can be either insideor outside of the vehicle compartment. The particular design of thelaser radar assemblies 262 and 263 are similar to those used in FIGS.21A, 21B, 22A and 22B. Although not shown, other geometries are ofcourse possible such as having the laser radar assemblies mounted on ornear the roof for the rear assemblies and above the headlights for thefrontal assemblies or vice versa. Also, although assemblies mounted onthe corners of the vehicle are illustrated, in some cases it may bedesirable to mount laser radar assemblies in the center of the front,back and sides of the vehicle or a combination or center andcorner-mounted laser radar assemblies can be used.

FIG. 24 is a schematic illustration of a typical laser radar assemblyshowing the scanning or pointing system with simplified optics forillustration only. In an actual design, the optics will typicallyinclude multiple lenses. Also, the focal point will typically not beoutside of the laser radar assembly. In this non-limiting example, acommon optical system 267 is used to control a laser light 265 and animager or camera 266. The laser source transmits, usually eye-safeinfrared, light through its optical sub-system 271 which collimates theradiation. The collimated radiation is then reflected off mirror 273 tomirror 274 which reflects the radiation to the desired direction throughlens system 267. The direction of the beam can be controlled by a motor272 which can rotate both mirror 274 and optical system 267 to achievethe desired scanning or pointing angle for scanning designs. A preferredapproach is to use a non-scanning approach. The radiation leaves theoptical system 267 and illuminates the desired object or target 276. Theradiation reflected from object 276 can pass back through lens 267,reflect off mirror 274 pass through semitransparent minor 273 throughoptic subsystem 268 and onto optical sensitive surface 266. Many otherconfigurations are possible. The transmission of the radiation iscontrolled by optical shutter 270 via controller 275. Similarly, thelight that reaches the imager 266 is controlled by controller 275 andoptical shutter 269. These optical shutters 269, 270 can be liquidcrystal devices, Kerr or Pockel cells, garnet films, other spatial lightmonitors or, preferably, high speed optical shutters such as describedin patents and patent applications of the 3DV Systems Ltd., of Yokneam,Israel, as set forth above or equivalent. Since much of the technologyused in this invention related to the camera and shutter system isdisclosed in the 3DV patents and patent applications, it will not berepeated here, by is incorporated by reference herein.

In some embodiments, it may be important to assure that the lens throughwhich the laser radar radiation passes is clean. As a minimum, adiagnostic system is required to inform the RtZF® or other system thatthe lens are soiled and therefore the laser radar system can not berelied upon. Additionally, in some applications, means are provided toclean one or more of the lens or to remove the soiled surface. In thelatter case, a roll of thin film can be provided which, upon thedetection of a spoiled lens, rolls up a portion of the film and therebyprovides a new clean surface. When the roll is used up it can bereplaced. Other systems provide one or more cleaning methods such as asmall wiper or the laser radar unit can move the lens into a cleaningstation. Many other methods are of course possible and the inventionhere is basically concerned with ascertaining that the lens is clean andif not informing the system of this fact and, in some cases, cleaning orremoving the soiled surface. The lens can also be coated with a coatingthat resists soiling as disclosed in U.S. Pat. No. 7,136,494, U.S. Pat.No. 6,991,339 and U.S. Pat. No. 6,193,378.

Note that although laser radar and radar have been discussed separately,in some implementations, it is desirable to use both a radar system anda laser radar system. Such a case can be where the laser radar system isnot capable to achieve sufficient range in adverse weather whereas theradar has the requisite range but insufficient resolution. The radarunit can provide a warning that a potentially dangerous situation existsand thus the vehicle speed should be reduced until the laser radardevice and obtain an image with sufficient resolution to permit anassessment of the extent of the danger and determine whether appropriateactions should be undertaken.

6. Infrastructure-to-Vehicle Communication

6.1 General

In order to eliminate fatalities on roads and mitigate congestion, it iscritical that vehicles communicate with each other. The type ofcommunication can take at least two forms, that which is time criticalsuch when two vehicles are about to collide and that which can have somedelay such as information that the road is icy 2 miles ahead. Timecritical communication is discussed above. This section will concentrateon the not time-critical communication which can also includeinformation from a vehicle that passed through an area an hour prior tothe subject vehicle or information derived from a server that may not benear the vehicle. Thus, this second type of communications can involvean entity that is not a vehicle such as a network server. In many cases,such a server will be required such as when a vehicle transmits apicture of an accident that needs to be interpreted before it can beadded as a temporary update to a map of the area.

Referring to FIG. 26 to explain this multi-form of communications, amethod for transmitting information to a host vehicle traveling on aroad using two different types or ways of communications in accordancewith the invention includes generating information from one or moresources thereof to be wirelessly transmitted to an information receivingsystem resident on the host vehicle during travel of the vehicle 280.The sources may be other vehicles on the road(s) on which the vehicle istraveling or about to or expected to travel, or infrastructurefacilities, e.g., stations or transmitters. Thus, the information may beabout one or more roads on which the host vehicle will travel in thefuture from other vehicles which traveled the road prior to the hostvehicle.

The information is then prioritized to distinguish between highpriority, time-critical information of immediate relevance to operationof the vehicle and low priority, non-time-critical information ofnon-immediate relevance to the operation of the host vehicle 282. Thisprioritization may be performed by the information receiving systemresident on the vehicle, e.g., based on an initial transmission fromeach source, or at a data storage facility separate and apart from thehost vehicle at which the information is being gathered. Theprioritization may be performed based on the current position of thehost vehicle, the location of the source and/or identity of the source.Some sources can always be considered high priority sources, e.g.,vehicles within a pre-determined range and in an expected path of travelof the host vehicle.

In particular when prioritization is performed by the informationreceiving system resident on the vehicle, it can be performed using themethod described above with reference to FIG. 20 to prioritize thereceived information in the form of waves or signals, i.e., filtertransmissions from transmitters. That is, any transmission from aparticular transmitter deemed to be a transmission of interest (based ondecoding of the initial part of the transmission 252 a) may beconsidered high priority information whereas any transmission from atransmitter not deemed to contain information of interest (based ondecoding of the initial part of the transmission 252 a), is consideredlow priority information.

High priority information 284, such as information from vehicles inclose proximity to the host vehicle and information potentially usefulor necessary for collision avoidance, is preferably transmitted directlyfrom the source 286. This ensures that the host vehicle will immediatelyhave information necessary for it to continue safe operation of thevehicle, e.g., by avoiding collisions with other proximate vehicles orinfrastructure.

Low priority information 288, or any other information not deemed highpriority, is gathered at the data storage facility 290 and directedtherefrom to the host vehicle using the ubiquitous network describedbelow, e.g., the Internet 292.

6.2 Summary

FIG. 27 shows a schematic of the flow of data. Information to bewirelessly transmitted, preferably via a ubiquitous network, to aninformation receiving system resident on the “host” vehicle 294 duringtravel of the vehicle 294 is generated by one or more informationsources which include “probe” vehicles 294, traffic cameras 296 and roadsensors 298. The probe vehicles 294 provide information about one ormore roads on which the host vehicle will travel or is expected totravel at some time in the future, the difference being if the road thevehicle expects to travel on is congested, the driver of the hostvehicle can take an alternative route. Other sources of informationinclude data channels with weather information, i.e., meteorologicalreports, and traffic information such as that provided by highway,bridge and tunnel operators and municipalities. It is important to notethat the host vehicle can also be a probe vehicle, in that informationit obtains can be used for transmission to vehicles behind it on thesame path, and that a probe vehicle can be a host vehicle in thatinformation it receives was obtained by vehicle in front of it on thesame path. As such, FIG. 27 shows element 294 designated as vehicles.

This information is sent from the various sources, preferably over aubiquitous network, and is gathered in a central data storage,monitoring and/or processing facility 300, e.g., a network server ormainframe computer, which may entail directing the information sourcesto respond to inquiries for information from the data facility orprogramming the information sources to automatically provide theinformation at set times. The probe vehicles 294 can also continuallyprovide information limited only by the components of the transmissionunit thereon. The data facility 300 can also be programmed toautomatically access data channels on a regular basis to obtain currentinformation about roads and weather. Although the data facility 300gathers a large amount of information, not all of the information willbe directed to the vehicle 294, i.e., only potential relevantinformation will be considered for each vehicle 294 in communicationwith the data facility 300. Thus, different subsets of the totalavailable information will be generated for each host vehicle 294.

The data facility 300 includes software and hardware components whichenables it to prioritize the information to distinguish between highpriority, time-critical information of immediate relevance to operationof the host vehicle 294 and low priority, non-time-critical informationof non-immediate relevance to the operation of the host vehicle 294. Itcan thus be programmed to control and communicate with the informationreceiving system to cause it to receive and process high priorityinformation before low priority information, the transmission of both ofwhich are directed by the data facility 300. Prioritization can beestablished based on the current position of the host vehicle 294.

Data facility 300 can be programmed to maintain a map of roads residentin host vehicles by transmitting map updates necessary for the maps tobe current, the map updates being generated based on the gatheredinformation. If a temporary map update is created based on a change inthe operability or functionality of a road, e.g., based on a trafficaccident, the data facility 300 is programmed to continuously monitorthe change to determine when the use of the road reverts to a statepreceding the change. When this happens, notification of this reversionis transmitted to the host vehicle, e.g., via another map update.

Data facility 300 communicates with traffic control devices 302 via theubiquitous network of transceivers. It can thus analyze vehiculartraffic and control the traffic control devices based on the vehiculartraffic, e.g., regulate the pattern of green lights to optimize traffic,eliminate traffic jams and expedite emergency response vehicles.

Data facility 300 also communicates with an emergency response facility304 to direct aid to a host vehicle when necessary or to the site of anaccident as determined by the information gathered from the sourcesthereof.

Data facility 300 also communications with Internet content providers306 to allow the occupants of host vehicles to request Internet contentover the ubiquitous network.

It should be understood that the transmission of information betweenvehicles is one exemplifying use of the invention which also encompassesgenerating information from other types of mobile units, transmittingthe information to a common monitoring station, generating at themonitoring station an update for, e.g., a map, based on the transmittedinformation, and then transmitting the update to each of the mobileunits.

7. The RtZF® System

7.1 Road Departure

FIG. 4 is a logic diagram of the system 50 in accordance with theinvention showing the combination 40 of the GPS and DGPS processingsystems 42 and an inertial reference unit (IRU) or inertial navigationsystem (INS) or Inertial Measurement Unit (IMU) 44. The GPS systemincludes a unit for processing the received information from thesatellites 2 of the GPS satellite system, the information from thesatellites 30 of the DGPS system and data from the inertial referenceunit 44. The inertial reference unit 44 contains accelerometers andlaser or MEMS gyroscopes.

The system shown in FIG. 4 is a minimal RtZF® system that can be used toprevent road departure, lane crossing and intersection accidents, whichtogether account for more than about 50% of the fatal accidents in theU.S.

Map database 48 works in conjunction with a navigation system 46 toprovide a warning to the driver when he or she is about to run off theroad, cross a center (yellow) line, run a stop sign, or run a redstoplight. The map database 48 contains a map of the roadway to anaccuracy of 2 cm (1σ), i.e., data on the edges of the lanes of theroadway and the edges of the roadway, and the location of all stop signsand stoplights and other traffic control devices such as other types ofroad signs. Another sensor, not shown, provides input to the vehicleindicating that an approaching stoplight is red, yellow or green.Navigation system 46 is coupled to the GPS and DGPS processing system42. For this simple system, the driver is warned if any of the aboveevents is detected by a driver warning system 45 coupled to thenavigation system 46. The driver warning system 45 can be an alarm,light, buzzer or other audible noise, or, preferably, a simulated rumblestrip for yellow line and “running off of road” situations and acombined light and alarm for the stop sign and stoplight infractions.

7.2 Accident Avoidance

FIG. 5 is a block diagram of the more advanced accident avoidance systemof this invention and method of the present invention illustratingsystem sensors, transceivers, computers, displays, input and outputdevices and other key elements.

As illustrated in FIG. 5, the vehicle accident avoidance system isimplemented using a variety of microprocessors and electronic circuits100 to interconnect and route various signals between and among theillustrated subsystems. GPS receiver 52 is used to receive GPS radiosignals as illustrated in FIG. 1. DGPS receiver 54 receives thedifferential correction signals from one or more base stations eitherdirectly or via a geocentric stationary or LEO satellite, an earth-basedstation or other means. Inter-vehicle communication subsystem 56 is usedto transmit and receive information between various nearby vehicles.This communication will in general take place via broadband orultra-broadband communication techniques, or on dedicated frequencyradio channels, or in a preferred mode, noise communication system asdescribed above. This communication may be implemented using multipleaccess communication methods including frequency division multipleaccess (FDMA), time division multiple access (TDMA), or code divisionmultiple access (CDMA), or noise communication system, in a manner topermit simultaneous communication with and between vehicles. Other formsof communication between vehicles are possible such as through theInternet. This communication may include such information as the preciselocation of a vehicle, the latest received signals from the GPSsatellites in view, other road condition information, emergency signals,hazard warnings, vehicle velocity and intended path, and any otherinformation which is useful to improve the safety of the vehicle roadsystem.

Infrastructure communication system 58 permits bi-directionalcommunication between the host vehicle and the infrastructure andincludes such information transfer as updates to the digital maps,weather information, road condition information, hazard information,congestion information, temporary signs and warnings, and any otherinformation which can improve the safety of the vehicle highway system.

Cameras 60 are used generally for interrogating environment nearby thehost vehicle for such functions as blind spot monitoring, backupwarnings, anticipatory crash sensing, visibility determination, lanefollowing, and any other visual information which is desirable forimproving the safety of the vehicle highway system. Generally, thecameras will be sensitive to infrared and/or visible light, however, insome cases a passive infrared camera will the used to detect thepresence of animate bodies such as deer or people on the roadway infront of the vehicle. Frequently, infrared or visible illumination willbe provided by the host vehicle. In the preferred system, highbrightness eye-safe IR will be used.

Radar 62 is primarily used to scan an environment close to and furtherfrom the vehicle than the range of the cameras and to provide an initialwarning of potential obstacles in the path of the vehicle. The radar 62can also be used when conditions of a reduced visibility are present toprovide advance warning to the vehicle of obstacles hidden by rain, fog,snow etc. Pulsed, continuous wave, noise or micropower impulse radarsystems can be used as appropriate. Also, Doppler radar principles canbe used to determine the object to host vehicle relative velocity.

Laser or terahertz radar 64 is primarily used to illuminate potentialhazardous objects in the path of the vehicle. Since the vehicle will beoperating on accurate mapped roads, the precise location of objectsdiscovered by the radar or camera systems can be determined using rangegating and scanning laser radar as described above or by phasetechniques.

The driver warning system 66 provides visual and/or audible warningmessages to the driver or others that a hazard exists. In addition toactivating a warning system within the vehicle, this system can activatesound and/or light systems to warn other people, animals, or vehicles ofa pending hazardous condition. In such cases, the warning system couldactivate the vehicle headlights, tail lights, horn and/or thevehicle-to-vehicle, Internet or infrastructure communication system toinform other vehicles, a traffic control station or other base station.This system will be important during the early stages of implementationof RtZF®, however as more and more vehicles are equipped with thesystem, there will be less need to warn the driver or others ofpotential problems.

Map database subsystem 68, which could reside on an external memorymodule, will contain all of the map information such as road edges up to2 cm accuracy, the locations of stop signs, stoplights, lane markersetc. as described above. The fundamental map data can be organized onread-only magnetic or optical memory with a read/write associated memoryfor storing map update information. Alternatively, the map informationcan be stored on rewritable media that can be updated with informationfrom the infrastructure communication subsystem 58. This updating cantake place while the vehicle is being operated or, alternatively, whilethe vehicle is parked in a garage or on the street.

Three servos are provided for controlling the vehicle during the laterstages of implementation of the RtZF® product and include a brake servo70, a steering servo 72, and a throttle servo 74. The vehicle can becontrolled using deterministic, fuzzy logic, neural network or,preferably, neural-fuzzy algorithms.

As a check on the inertial system, a velocity sensor 76 based on a wheelspeed sensor, or ground speed monitoring system using lasers, radar orultrasonics, for example, can be provided for the system. A radarvelocity meter is a device which transmits a noise modulated radar pulsetoward the ground at an angle to the vertical and measures the Dopplervelocity of the returned signal to provide an accurate measure of thevehicle velocity relative to the ground. Another radar device can bedesigned which measures the displacement of the vehicle. Othermodulation techniques and other radar systems can be used to achievesimilar results. Other systems are preferably used for this purpose suchas the GPS/DGPS or precise position systems.

The inertial navigation system (INS), sometimes called the inertialreference unit or IRU, comprises one or more accelerometers 78 and oneor more gyroscopes 80. Usually, three accelerometers would be requiredto provide the vehicle acceleration in the latitude, longitude andvertical directions and three gyroscopes would be required to providethe angular rate about the pitch, yaw and roll axes. In general, agyroscope would measure the angular rate or angular velocity. Angularacceleration may be obtained by differentiating the angular rate.

A gyroscope 80, as used herein in the IRU, includes all kinds ofgyroscopes such as MEMS-based gyroscopes, fiber optic gyroscopes (FOG)and accelerometer-based gyroscopes.

Accelerometer-based gyroscopes encompass a situation where twoaccelerometers are placed apart and the difference in the accelerationis used to determine angular acceleration and a situation where anaccelerometer is placed on a vibrating structure and the Coriolis effectis used to obtain the angular velocity.

The possibility of an accelerometer-based gyroscope 80 in the IRU ismade possible by construction of a suitable gyroscope by InterstateElectronics Corporation (IEC). IEC manufactures IMUs in volume based onμSCIRAS (micro-machined Silicon Coriolis Inertial Rate and AccelerationSensor) accelerometers. Detailed information about this device can befound at the IEC website at iechome.com.

An accelerometer 78, as used herein in the IRU, includes conventionalpiezoelectric-based accelerometers, MEMS-based accelerometers (such asmade by Analog Devices) and the type as described in U.S. Pat. No.6,182,509.

Display subsystem 82 includes an appropriate display driver and either aheads-up or other display system for providing system information to thevehicle operator. Display subsystem 82 may include multiple displays fora single occupant or for multiple occupants, e.g., directed towardmultiple seating positions in the vehicle. One type of display may be adisplay made from organic light emitting diodes (OLEDs). Such a displaycan be sandwiched between the layers of glass that make up thewindshield and does not require a projection system.

The information being displayed on the display can be in the form ofnon-critical information such as the location of the vehicle on a map,as selected by the vehicle operator and/or it can include warning orother emergency messages provided by the vehicle subsystems or fromcommunication with other vehicles or the infrastructure. An emergencymessage that the road has been washed out ahead, for example, would bean example of such a message.

Generally, the display will make use of icons when the position of thehost vehicle relative to obstacles or other vehicles is displayed.Occasionally, as the image can be displayed especially when the objectcannot be identified. Icons can be selected which are representative ofthe transmitters from which wireless signals are received.

A general memory unit 84 which can comprise read-only memory or randomaccess memory or any combination thereof, is shown. This memory module,which can be either located at one place or distributed throughout thesystem, supplies the information storage capability for the system.

For advanced RtZF® systems containing the precise positioningcapability, subsystem 86 provides the capability of sending andreceiving information to infrastructure-based precise positioning tagsor devices which may be based on noise or micropower impulse radartechnology, IR lasers, radar or IR reflector (e.g. corner cube ordihedral) or RFIR technology or equivalent. Once again the PPS systemcan also be based on a signature analysis using the adaptive associativememory technology or equivalent.

In some locations where weather conditions can deteriorate and degraderoad surface conditions, various infrastructure-based sensors can beplaced either in or adjacent to the road surface. Subsystem 88 isdesigned to interrogate and obtained information from such road-basedsystems. An example of such a system would be an RFID tag containing atemperature sensor. This device may be battery-powered or, preferably,would receive its power from energy harvesting (e.g., solar energy,vibratory energy), the vehicle-mounted interrogator, or other hostvehicle-mounted source, as the vehicle passes nearby the device. In thismanner, the vehicle can obtain the temperature of the road surface andreceive advanced warning when the temperature is approaching conditionswhich could cause icing of the roadway, for example. An RFID based on asurface acoustic wave (SAW) device is one preferred example of such asensor, see U.S. Pat. No. 6,662,642. An infrared sensor on the vehiclecan also be used to determine the road temperature and the existence ofice or snow.

For some implementations of the RtZF® system, stoplights will be fittedwith transmitters which will broadcast a signal when the light is red.Such a system could make use of the vehicle noise communication systemas described above. This signal can be then received by a vehicle thatis approaching the stoplight provided that vehicle has the proper sensoras shown as 92. Alternatively, a camera can be aimed in the direction ofstoplights and, since the existence of the stoplight will be known bythe system, as it will have been recorded on the map, the vehicle willknow when to look for a stoplight and determine the color of the light.

An alternative idea is for the vehicle to broadcast a signal to thestoplight if, via a camera or other means, it determines that the lightis red. If there are no vehicles coming from the other direction, thelight can change permitting the vehicle to proceed without stopping.Similarly, if the stoplight has a camera, it can look in all directionsand control the light color depending on the number of vehiclesapproaching from each direction. A system of phasing vehicles can alsobe devised whereby the speed of approaching vehicles is controlled sothat they interleave through the intersection and the stoplight may notbe necessary.

Although atomic clocks are probably too expensive to the deployed onautomobiles, nevertheless there has been significant advances recentlyin the accuracy of clocks to the extent that it is now feasible to placea reasonably accurate clock as a subsystem 94 to this system. Since theclock can be recalibrated from each DGPS transmission, the clock driftcan be accurately measured and used to predict the precise time eventhough the clock by itself may be incapable of doing so. To the extentthat the vehicle contains an accurate time source, the satellites inview requirement can temporarily drop from 4 to 3. An accurate clockalso facilitates the carrier phase DGPS implementations of the system asdiscussed above. Additionally, as long as a vehicle knows approximatelywhere it is on the roadway, it will know its altitude from the map andthus one less satellite is necessary.

Power is supplied to the system as shown by power subsystem 96. Certainoperator controls are also permitted as illustrated in subsystem 98.

The control processor or central processor and circuit board subsystem100 to which all of the above components 52-98 are coupled, performssuch functions as GPS ranging, DGPS corrections, image analysis, radaranalysis, laser radar scanning control and analysis of receivedinformation, warning message generation, map communication, vehiclecontrol, inertial navigation system calibrations and control, displaycontrol, precise positioning calculations, road condition predictions,and all other functions needed for the system to operate according todesign.

A display could be provided for generating and displaying warningmessages which is visible to the driver and/or passengers of thevehicle. The warning could also be in the form of an audible tone, asimulated rumble strip and light and other similar ways to attract theattention of the driver and/or passengers. Although vibration systemshave been proposed by others, the inventors have found that a pure noiserumble strip is preferred and is simpler and less costly to implement,

Vehicle control also encompasses control over the vehicle to preventaccidents. By considering information from the map database 48, from thenavigation system 46, and the position of the vehicle obtained via GPS,DGPS and PPS systems, a determination can be made whether the vehicle isabout to run off the road, cross a yellow line and run a stop sign, aswell as the existence or foreseen occurrence of other potential crashsituations. The color of an approaching stoplight can also be factoredin the vehicle control as can information from the vehicle to vehicle,vehicle to infrastructure and around vehicle radar, visual or IRmonitoring systems.

FIG. 5A shows a selected reduced embodiment of the accident avoidancesystem shown in FIG. 5. The system includes an inertial reference unitincluding a plurality of accelerometers and gyroscopes, namelyaccelerometers 78A, preferably three of any type disclosed above, andgyroscopes 80A, preferably three of any type disclosed above. Anaccurate clock 94A is provided to obtain a time base or time reference.This system will accurately determine the motion (displacement,acceleration and/or velocity) of the vehicle in 6 degrees of freedom (3displacements (longitudinal, lateral and vertical)) via theaccelerometers 78A and three rotations (pitch, yaw and roll) via thegyroscopes 80A. As such, along with a time base from clock 94A, theprocessor 100A can determine that there was an accident and preciselywhat type of accident it was in terms of the motion of the vehicle(frontal, side, rear and rollover). This system is different from acrash sensor in that this system can reside in the passenger compartmentof the vehicle where it is protected from actually being in the accidentcrush and/or crash zones and thus it does not have to forecast theaccident severity. It knows the resulting vehicle motion and thereforeexactly what the accident was and what the injury potential is. Atypical crash sensor can get destroyed or at least rotated during thecrash and thus will not determine the real severity of the accident.

Processor 100A is coupled to the inertial reference unit and also iscapable of performing the functions of vehicle control, such as viacontrol of the brake system 70A, steering system 72A and throttle system74A, crash sensing, rollover sensing, cassis control sensing, navigationfunctions and accident prevention as discussed herein.

Preferably, a Kalman filter is used to optimize the data from theinertial reference unit as well as other input sources of data, signalsor information. Also, a neural network, fuzzy logic or neural-fuzzysystem could be used to reduce the data obtained from the varioussensors to a manageable and optimal set. The actual manner in which aKalman filter can be constructed and used in the invention would be leftto one skilled in the art. Note that in the system of the inventionsdisclosed herein, the extensive calibration process carried on by othersuppliers of inertial sensors is not required since the systemperiodically corrects the errors in the sensors and revises thecalibration equation. This in some cases can reduce the manufacturingcost on the IMU by a factor of ten.

Further, the information from the accelerometers 78A and gyroscopes 80Ain conjunction with the time base or reference is transmittable via thecommunication system 56A,58A to other vehicles, possibly for the purposeof enabling other vehicles to avoid accidents with the host vehicle,and/or to infrastructure.

One particularly useful function would be for the processor to send datafrom, or data derived from, the accelerometers and gyroscopes relatingto a crash, i.e., indicative of the severity of the accident with thepotential for injury to occupants, to a monitoring location for thedispatch of emergency response personnel, i.e., an EMS facility or firestation. Other telematics functions could also be provided.

7.3 Exterior Surveillance System

FIG. 6 is a block diagram of the host vehicle exterior surveillancesystem. Cameras 60 are primarily intended for observing the immediateenvironment of the vehicle. They are used for recognizing objects thatcould be most threatening to the vehicle, i.e., closest to the vehicle.These objects include vehicles or other objects that are in the vehicleblind spot, objects or vehicles that are about to impact the hostvehicle from any direction, and objects either in front of or behind thehost vehicle which the host vehicle is about to impact. These functionsare normally called blind spot monitoring and collision anticipatorysensors.

As discussed above, the cameras 60 can use naturally occurring visibleor infrared radiation (particularly eye-safe IR), or other parts of theelectromagnetic spectrum including terahertz and x-rays, or they may besupplemented with sources of visible or infrared illumination from thehost vehicle. Note that there generally is little naturally occurringterahertz radiation other than the amount that occurs in black bodyradiation from all sources. The cameras 60 used are preferably highdynamic range cameras that have a dynamic range exceeding 60 db andpreferably exceeding 100 db. Such commercially available cameras includethose manufactured by the Photobit Corporation in California and the IMSChips Company in Stuttgart Germany. Alternately, various other meansexist for increasing the effective dynamic range through shutter controlor illumination control using a Kerr or Pokel cell, modulatedillumination, external pixel integration etc.

These cameras are based on CMOS technology and can have the importantproperty that pixels are independently addressable. Thus, the controlprocessor may decide which pixels are to be read at a particular time.This permits the system to concentrate on certain objects of interestand thereby make more effective use of the available bandwidth.

Video processor printed circuit boards or feature extractor 61 can belocated adjacent and coupled to the cameras 60 so as to reduce theinformation transferred to the control processor. The video processorboards or feature extractor 61 can also perform the function of featureextraction so that all values of all pixels do not need to be sent tothe neural network for identification processing. The feature extractionincludes such tasks as determining the edges of objects in the sceneand, in particular, comparing and subtracting one scene from another toeliminate unimportant background images and to concentrate on thoseobjects which had been illuminated with infrared or terahertz radiation,for example, from the host vehicle. By these and other techniques, theamount of information to be transferred to the neural network issubstantially reduced.

The neural network 63 receives the feature data extracted from thecamera images by the video processor feature extractor 61 and uses thisdata to determine the identification of the object in the image. Theneural network 63 has been previously trained on a library of imagesthat can involve as many as one million such images. Fortunately, theimages seen from one vehicle are substantially the same as those seenfrom another vehicle and thus the neural network 63 in general does notneed to be trained for each type of host vehicle.

As the number of image types increases, modular or combination neuralnetworks can be used to simplify the system.

Although the neural network 63 has in particular been described, otherpattern recognition techniques are also applicable. One such techniqueuses the Fourier transform of the image and utilizes either opticalcorrelation techniques or a neural network trained on the Fouriertransforms of the images rather than on the image itself. In one case,the optical correlation is accomplished purely optically wherein theFourier transform of the image is accomplished using diffractiontechniques and projected onto a display, such as a garnet crystaldisplay, while a library of the object Fourier transforms is alsodisplayed on the display. By comparing the total light passing throughthe display, an optical correlation can be obtained very rapidly.Although such a technique has been applied to scene scanning by militaryhelicopters, it has previously not been used in automotive applications.

The laser radar system 64 is typically used in conjunction with ascanner 65. The scanner 65 typically includes two oscillating mirrors,or a MEMS mirror capable of oscillating in two dimensions, which causethe laser light to scan the two dimensional angular field. Alternately,the scanner can be a solid-state device utilizing a crystal having ahigh index of refraction which is driven by an ultrasonic vibrator asdiscussed above or rotating mirrors. The ultrasonic vibrator establisheselastic waves in the crystal which diffracts and changes the directionof the laser light. Another method is to use the DLP technology fromTexas Instruments. This technology allows more than 1 million MEMSminors to control the direction of the laser light.

The laser beam can be frequency, amplitude, time, code or noisemodulated so that the distance to the object reflecting the light can bedetermined. The laser light strikes an object and is reflected backwhere it can be guided onto a pin diode, or other high speed photodetector. Since the direction of laser light is known, the angularlocation of the reflected object is also known and since the laser lightis modulated the distance to the reflected point can be determined. Byvarying modulation frequency of the laser light, or through noise orcode modulation, the distance can be very precisely measured.

Alternatively, the time-of-flight of a short burst of laser light can bemeasured providing a direct reading of the distance to the object thatreflected the light. By either technique, a three-dimensional map can bemade of the surface of the reflecting object. Objects within a certainrange of the host vehicle can be easily separated out using the rangeinformation. This can be done electronically using a technique calledrange gating, or it can be accomplished mathematically based on therange data. By this technique, an image of an object can be easilyseparated from other objects based on distance from the host vehicle.

Since the vehicle knows its position accurately and in particular itknows the lane on which it is driving, a determination can be made ofthe location of any reflective object and in particular whether or notthe reflective object is on the same lane as the host vehicle. This factcan be determined since the host vehicle has a map and the reflectiveobject can be virtually placed on that map to determine its location onthe roadway, for example.

The laser radar system will generally operate in the near-infrared partof the electromagnetic spectrum and preferably in the eye-safe part. Thelaser beam will be of relatively high intensity compared to thesurrounding radiation and thus even in conditions of fog, snow, andheavy rain, the penetration of the laser beam and its reflection willpermit somewhat greater distance observations than the human driver canperceive. Under the RtZF® plan, it is recommended that the speed of thehost vehicle be limited such that vehicle can come to a complete stop inone half or less of the visibility distance. This will permit the laserradar system to observe and identify threatening objects that are beyondthe visibility distance, apply the brakes to the vehicle if necessarycausing the vehicle to stop prior to an impact, providing an addeddegree of safety to the host vehicle.

Radar system 62 is mainly provided to supplement laser radar system. Itis particularly useful for low visibility situations where thepenetration of the laser radar system is limited. The radar system,which is most probably a noise or pseudonoise modulated continuous waveradar, can also be used to provide a crude map of objects surroundingthe vehicle. The most common use for automotive radar systems is foradaptive cruise control systems where the radar monitors the distanceand, in some cases, the velocity of the vehicle immediately in front ofthe host vehicle. The radar system 62 is controlled by the controlprocessor 100.

Display system 82 was discussed previously and can be either a heads upor other appropriate display.

Control processor 100 can be attached to a vehicle special or generalpurpose bus 110 for transferring other information to and from thecontrol processor to other vehicle subsystems.

In interrogating other vehicles on the roadway, a positiveidentification of the vehicle and thus its expected properties such asits size and mass can sometimes be accomplished by laser vibrometry. Bythis method, a reflected electromagnetic wave can be modulated based onthe vibration that the vehicle is undergoing. Since this vibration iscaused at least partially by the engine, and each class of engine has adifferent vibration signature, this information can be used to identifythe engine type and thus the vehicle. This technique is similar to oneused to identify enemy military vehicles by the U.S. military. It isalso used to identify ships at sea using hydrophones. In the presentcase, a laser beam is directed at the vehicle of interest and thereturned reflected beam is analyzed such as with a Fourier transform todetermine the frequency makeup of the beam. This can then be related toa vehicle to identify its type either through the use of a look-up tableor neural network or other appropriate method. This information can thenbe used as information in connection with an anticipatory sensor as itwould permit a more accurate estimation of the mass of a potentiallyimpacting vehicle.

Once the vehicle knows where it is located, this information can bedisplayed on a heads-up display and if an occupant sensor has determinedthe location of the eyes of the driver, the road edges, for example, andother pertinent information from the map database can be displayedexactly where they would be seen by the driver. For the case of drivingin dense fog or on a snow covered road, the driver will be able to seethe road edges perhaps exactly or even better than the real view, insome cases. Additionally, other information gleaned by the exteriormonitoring system can show the operator the presence of other vehiclesand whether they represent a threat to the host vehicle (see for example“Seeing the road ahead”, GPS World Nov. 1, 2003, which describes asystem incorporating many of the current assignee's ideas describedherein).

7.4 Corridors

FIG. 7 shows an implementation of the invention in which a vehicle 18 istraveling on a roadway in a defined corridor in the direction X. Eachcorridor is defined by lines 14. If the vehicle is traveling in onecorridor and strays in the direction Y so that it moves along the line22, e.g., the driver is falling asleep, the system on board the vehiclein accordance with the invention will activate a warning. Morespecifically, the system continually detects the position of thevehicle, such as by means of the GPS, DGPS and/or PPS, and has thelocations of the lines 14 defining the corridor recorded in its mapdatabase. Upon an intersection of the position of the vehicle and one ofthe lines 14 as determined by a processor, the system may be designed tosound an alarm to alert the driver to the deviation or possibly evencorrect the steering of the vehicle to return the vehicle to within thecorridor defined by lines 14.

FIG. 8 shows an implementation of the invention in which a pair ofvehicles 18, 26 is traveling on a roadway each in a defined corridordefined by lines 14 and each is equipped with a system in accordancewith the invention. The system in each vehicle 18, 26 will receive datainforming it of the position of the other vehicle and prevent accidentsfrom occurring, e.g., if vehicle 18 moves in the direction of arrow 20.This can be accomplished via direct wireless broadband communication orany of the other communication methods described above, or throughanother path such as via the Internet or through a base station, whereineach vehicle transmits its best estimate of its absolute location on theearth along with an estimate of the accuracy of this location. If onevehicle has recently passed a precise positioning station, for example,then it will know its position very accurately to within a fewcentimeters. Each vehicle can also send the latest satellite messages,or a portion thereof or data derived therefrom, that it received,permitting each vehicle to precisely determine its relative location tothe other since the errors in the signals will be the same for bothvehicles. To the extent that both vehicles are near each other, even thecarrier phase ambiguity can be determined and each vehicle will know itsposition relative to the other to within better than a few centimeters.As more and more vehicles become part of the community and communicatetheir information to each other, each vehicle can even more accuratelydetermine its absolute position and especially if one vehicle knows itsposition very accurately, if it recently passed a PPS for example, thenall vehicles will know their position with approximately the sameaccuracy and that accuracy will be able to be maintained for as long asa vehicle keeps its lock on the satellites in view. If that lock is losttemporarily, the INS system will fill in the gaps and, depending on theaccuracy of that system, the approximate 2 centimeter accuracy can bemaintained even if the satellite lock is lost for up to approximatelyfive minutes.

A five minute loss of satellite lock is unlikely except in tunnels or inlocations where buildings or geological features interfere with thesignals. In the building case, the problem can be eliminated through theplacement of PPS stations, or through environmental signature analysis,and the same would be true for the geological obstruction case except inremote areas where ultra precise positioning accuracy is probably notrequired. In the case of tunnels, for example, the cost of adding PPSstations is insignificant compared with the cost of building andmaintaining the tunnel.

7.5 Vehicle Control

FIG. 12 a is a flow chart of the method in accordance with theinvention. The absolute position of the vehicle is determined at 130,e.g., using a GPS, DGPS, PPS system, and compared to the edges of theroadway at 134, which is obtained from a memory unit 132. Based on thecomparison at 134, it is determined whether the absolute position of thevehicle is approaching close to or intersects an edge of the roadway at136. If not, then the position of the vehicle is again obtained, e.g.,at a set time interval thereafter, and the process continues. If yes, analarm and/or warning system will be activated and/or the system willtake control of the vehicle (at 140) to guide it to a shoulder of theroadway or other safe location.

FIG. 12 b is another flow chart of the method in accordance with theinvention similar to FIG. 12 a. Again the absolute position of thevehicle is determined at 130, e.g., using a GPS, DGPS, PPS system, andcompared to the location of a roadway yellow line at 142 (or possiblyanother line which indicates an edge of a lane of a roadway), which isobtained from a memory unit 132. Based on the comparison at 144, it isdetermined whether the absolute position of the vehicle is approachingclose to or intersects the yellow line 144. If not, then the position ofthe vehicle is again obtained, e.g., at a set time interval thereafter,and the process continues. If yes, an alarm will sound and/or the systemwill take control of the vehicle (at 146) to control the steering orguide it to a shoulder of the roadway or other safe location.

FIG. 12 c is another flow chart of the method in accordance with theinvention similar to FIG. 12 a. Again the absolute position of thevehicle is determined at 130, e.g., using a GPS, DGPS, PPS system, andcompared to the location of a roadway stoplight at 150, which isobtained from a memory unit 132. Based on the comparison at 150, it isdetermined whether the absolute position of the vehicle is approachingclose to a stoplight. If not, then the position of the vehicle is againobtained, e.g., at a set interval thereafter, and the process continues.If yes, a sensor determines whether the stoplight is red (e.g., acamera, transmission from stoplight) and if so, an alarm will soundand/or the system will take control of the vehicle (at 154) to controlthe brakes or guide it to a shoulder of the roadway or other safelocation. A similar flow chart can be now drawn by those skilled in theart for other conditions such as stop signs, vehicle speed control,collision avoidance etc.

7.6 Intersection Collision Avoidance

FIG. 13 illustrates an intersection of a major road 170 with a lesserroad 172. The road 170 has the right of way and stop signs 174 have beenplaced to control the traffic on the lesser road 172. Vehicles 18 and 26are proceeding on road 172 and vehicle 25 is proceeding on road 170. Avery common accident is caused when vehicle 18 ignores the stop sign 174and proceeds into the intersection where it is struck on the side byvehicle 25 or strikes vehicle 25 on the side.

Using the teachings of this invention, vehicle 18 will know of theexistence of the stop sign and if the operator attempts to proceedwithout stopping, the system will sound a warning and if that warning isnot heeded, the system will automatically bring the vehicle 18 to astop, preventing it from intruding into the intersection.

Another common accident is where vehicle 18 does in fact stop but thenproceeds forward without noticing vehicle 25 thereby causing anaccident. Since in the fully deployed RtZF® system, vehicle 18 will knowthrough the vehicle-to-vehicle communication the existence and locationof vehicle 25 and can calculate its velocity, the system can once againtake control of vehicle 18 if a warning is not heeded and preventvehicle 18 from proceeding into the intersection and thereby prevent theaccident.

In the event that the vehicle 25 is not equipped with the RtZF® system,vehicle 18 will still sense the presence of vehicle 25 through the laserradar, radar and camera systems. Once again, when the position andvelocity of vehicle 25 is sensed, appropriate action can be taken by thesystem in vehicle 18 to eliminate the accident.

In another scenario where vehicle 18 properly stops at the stop sign,but vehicle 26 proceeds without observing the presence of the stoppedvehicle 18, the laser radar, radar and camera systems will all operateto warn the driver of vehicle 26 and if that warning is not heeded, thesystem in vehicle 26 will automatically stop the vehicle 26 prior to itsimpacting vehicle 18. Thus, in the scenarios described above the “Roadto Zero Fatalities”® system and method of this invention will preventcommon intersection accidents from occurring.

FIG. 14 is a view of an intersection where traffic is controlled bystoplights 180. If the vehicle 18 does not respond in time to a redstoplight, the system as described above will issue a warning and if notheeded, the system will take control of the vehicle 18 to prevent itfrom entering the intersection and colliding with vehicle 25. In thiscase, the stoplight 180 will emit a signal indicating its color, such asby way of the communication system, and/or vehicle 18 will have a cameramounted such that it can observe the color of the stoplight. There areof course other information transfer methods such as through theInternet. In this case, buildings 182 obstruct the view from vehicle 18to vehicle 25 thus an accident can still be prevented even when theoperators are not able to visually see the threatening vehicle. If bothvehicles have the RtZF® system they will be communicating and theirpresence and relative positions will be known to both vehicles.

FIG. 15 illustrates the case where vehicle 18 is about to execute aleft-hand turn into the path of vehicle 25. This accident will beprevented if both cars have the RtZF® system since the locations andvelocities of both vehicles 18, 25 will be known to each other. Ifvehicle 25 is not equipped and vehicle 18 is, then the camera, radar,and laser radar subsystems will operate to prevent vehicle 18 fromturning into the path of vehicle 25. Once again common intersectionaccidents are prevented by this invention.

The systems described above can be augmented by infrastructure-basedsensing and warning systems. Camera, laser or terahertz radar or radarsubsystems such as placed on the vehicle can also be placed atintersections to warn the oncoming traffic if a collision is likely tooccur. Additionally, simple sensors that sense the signals emitted byoncoming vehicles, including radar, thermal radiation, etc., can be usedto operate warning systems that notify oncoming traffic of potentiallydangerous situations. Thus, many of the teachings of this invention canbe applied to infrastructure-based installations in addition to thevehicle-resident systems.

While the invention has been illustrated and described in detail in thedrawings and the foregoing description, the same is to be considered asillustrative and not restrictive in character, it being understood thatonly preferred embodiments have been shown and described and that allchanges and modifications that come within the spirit of the inventionare desired to be protected.

This application is one in a series of applications covering safety andother systems for vehicles and other uses. The disclosure herein goesbeyond that needed to support the claims of the particular inventionthat is claimed herein. This is not to be construed that the inventorsare thereby releasing the unclaimed disclosure and subject matter intothe public domain. Rather, it is intended that patent applications havebeen or will be filed to cover all of the subject matter disclosedabove.

1. A vehicular arrangement for obtaining information about objectsexterior to the vehicle, comprising: a plurality of distance measuringsystems fixed to the vehicle along the windshield, an edge or a side ofthe vehicle, each of said distance measuring systems including: a cameraconfigured to obtain images of objects exterior of the vehicle; aprocessor that determines a distance between the vehicle and objectsincluded in images obtained by said camera and a speed of the objectsbased on the successive distance determinations; and a reactive systemarranged on the vehicle and coupled to said distance measuring systemsand configured to consider both distance between the objects and thevehicle and the speed of the objects and react accordingly, and whereina position of said distance measuring systems is selected to encompassan area in front of the vehicle, an area behind the vehicle and an areaon each side of the vehicle in the fixed fields of view of said distancemeasuring systems.
 2. The arrangement of claim 1, wherein said reactivesystem is a system which limits the speed of the vehicle such that thespeed of the vehicle can be limited in the presence of other objects orvehicles within a threshold distance from the vehicle.
 3. Thearrangement of claim 1, wherein said reactive system is a warning systemfor providing a warning to a driver of the vehicle about the objectsexterior of the vehicle.
 4. The arrangement of claim 1, wherein saidreactive system is configured to access a map database containinginformation about vehicular travel lanes and consider the informationwhen determining how to react to the presence of the object.
 5. Thearrangement of claim 4, wherein said reactive system is configured to,based on the map database, ignore objects on travel lanes where there isa physical barrier separating these travel lanes from the travel lane onwhich the vehicle is traveling.
 6. The arrangement of claim 4, whereinsaid reactive system is configured to, based on the map database, ignorestationary objects on a side of the travel lane.
 7. The arrangement ofclaim 4, wherein said reactive system is configured to, based on the mapdatabase, differentiate between stationary objects on a side of thetravel lane and stationary objects in the travel lane.
 8. A vehicle,comprising: the arrangement of claim 1 having four of said distancemeasuring systems, one arranged at each of the four corners of thevehicle, and which are coupled to one or more common reactive systems.9. The vehicle of claim 8, wherein two of said distance measuringsystems are arranged above headlights of the vehicle.
 10. A vehicle,comprising: the arrangement of claim 1 having four of said distancemeasuring systems, one arranged at each of the four sides of thevehicle, and which are coupled to one or more common reactive systems.11. A vehicle, comprising: the arrangement of claim 1 having four ofsaid distance measuring systems, one of said four distance measuringsystems being located on or near a roof of the vehicle, and which arecoupled to one or more common reactive systems.
 12. A method foroperating a vehicle in consideration of objects exterior to the vehicle,comprising: obtaining images of objects exterior of the vehicle by meansof a camera of a plurality of distance measuring systems, each of thedistance measuring systems being fixed to the vehicle along thewindshield, an edge or a side of the vehicle, a position of the distancemeasuring systems encompassing an area in front of the vehicle, an areabehind the vehicle and an area on each side of the vehicle in the fixedfields of view of the distance measuring systems; determining using aprocessor, a distance between the vehicle and objects included in imagesobtained by the camera and a speed of the objects based on thesuccessive distance determinations; determining using the processor,whether an action must be undertaken to prevent contact between theobject and the vehicle based on the objects and their distance from thevehicle, and if so undertaking an action to prevent contact between thevehicle and the object.
 13. The method of claim 12, wherein the step ofundertaking an action comprises limiting the speed of the vehicle in thepresence of other vehicles determined to be within a threshold distancefrom the vehicle.
 14. The method of claim 12, wherein the step ofundertaking an action comprises providing a warning to a driver of thevehicle about the objects exterior of the vehicle.
 15. The method ofclaim 12, wherein the step of determining using the processor, whetheran action must be undertaken to prevent contact between the object andthe vehicle comprises accessing a map database containing informationabout vehicular travel lanes and considering the position of the objectrelative to a map in the map database.
 16. The method of claim 15,wherein the step of determining using the processor, whether an actionmust be undertaken to prevent contact between the object and the vehiclecomprises determining, based on the map database, whether there is aphysical barrier separating travel lanes on which the objects are movingfrom the travel lane on which the vehicle is moving, the method furthercomprising precluding the action from being undertaken when it isdetermined, based on the map database, that there is a physical barrierseparating these travel lanes from the travel lane on which the vehicleis moving.
 17. The method of claim 15, wherein the step of determiningusing the processor, whether an action must be undertaken to preventcontact between the object and the vehicle comprises determining, basedon the map database, whether the object is a stationary object on a sideof the travel lane, the method further comprising precluding the actionfrom being undertaken when it is determined, based on the map database,that the object is a stationary object on a side of the travel lane. 18.The method of claim 15, wherein the step of determining using theprocessor, whether an action must be undertaken to prevent contactbetween the object and the vehicle comprises configuring the system todifferentiate, based on the map database, between stationary objects ona side of the travel lane and stationary objects in the travel lane. 19.The method of claim 12, wherein four of the distance measuring systemsare arranged at each of the four corners of the vehicle.
 20. The methodof claim 12, wherein four of the distance measuring systems are arrangedon or near a roof of the vehicle.